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Executive summary 


The project 


The Parent Academy was a series of classes for pupils’ parents, designed to improve the English and 
mathematics attainment of pupils in Years 3 to 6 in English primary schools. Parents were offered the 
opportunity to participate in 12 Parent Academy classes, 6 on English and 6 on mathematics, delivered 
fortnightly by tutors with teaching qualifications and experience of teaching adults. The programme also 
included an educational family trip. 


The evaluation used a two-arm randomised controlled trial to test the efficacy of two versions of the 
intervention. In the first version, parents were incentivised to attend with a payment of £30 per session 
and in the second version they were not. Children of both groups of parents were compared with a 
similar group whose parents were not offered Parent Academy. Sixteen schools in two urban local 
authorities took part in the trial. A total of 2,593 children were involved. The project also included a 
process evaluation which assessed how the intervention was delivered and reported on its perceived 
benefits. The intervention was developed by the University of Chicago. It was not manualised and 
involved the development of a new adult learning course. The intervention and evaluation were funded 
by the Education Endowment Foundation and the KPMG Foundation. The trial took place between 
September 2014 and July 2015 with classes delivered between November 2014 and June 2015. 


Key Conclusions 


1. There is no evidence that offering free Parent Academy classes improved mathematics or reading 
outcomes for the children in the trial, even when parents were given a financial incentive to attend. 


2. In general, parental attendance at sessions was very low. However, even when the evaluators 
took this into account they found no evidence that parental attendance improved pupil outcomes. 


3. Offering financial incentives improved attendance at Parent Academy, suggesting this may be an 
effective way to engage and retain parents in interventions of this type. 


4. Ingeneral, evaluation participants felt that attending Parent Academy gave parents the confidence 
and skills to engage more effectively with their children’s learning. 


5. Staff reported that having a designated project lead in each school and using multiple methods to 
engage parents (rather than just written communication) were necessary for successful delivery. 
Schools need to consider these cost implications when deciding whether to adopt this intervention. 


Security rating awarded as 
part of the EEF peer review 


Findings from this study have moderate to high security. The study was designed as a three-arm 
randomised controlled trial which aimed to compare the progress of pupils whose parents were invited 
to participate in Parent Academy with those whose parents were not. The trial was an efficacy trial, 
which means that it tested the intervention under ideal, developer-led conditions. The trial was well 
designed and well powered, and the sample was well balanced on all observable characteristics. The 
proportion of pupils excluded from the analysis because they did not complete the tests at the end of 
the intervention was just over 10%. 


How secure are the findings? 


What are the findings? 


Table 1 below presents the main results. The evaluation did not find evidence that offering parents either 
incentivised or non-incentivised Parent Academy classes improved maths or reading outcomes for 
pupils in the trial. The estimated impact on each outcome was close to zero. This is true for the group 
of all pupils in Years 3 to 6 and for pupils eligible for free school meals. For children whose parents 
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attended the non-incentivised sessions, the estimated impacts on maths and English were actually 
negative, but these results may have been due to chance, so we would not conclude that Parent 
Academy has a negative impact. 


The intervention had a very low attendance rate. Six in ten parents (60%) who were offered the Parent 
Academy did not take part in any sessions and only around one in ten (11%) attended all twelve 
sessions. To assess whether the low attendance rates might account for the lack of impact, additional 
analysis was done taking account of how many sessions parents had attended. There was no evidence 
that the intervention had more impact for parents who attended more sessions. Although the intervention 
does not appear to have had an impact on pupil outcomes, there is evidence that the use of financial 
incentives did increase attendance at Parent Academy sessions. This suggests that financial incentives 
can be an effective way to encourage attendance at this type of intervention. 


This evaluation measured pupil outcomes within a few weeks of the intervention ending. Because the 
Parent Academy is designed to have an impact on pupils via changes in parent behaviour, it is possible 
that it would take longer than this for any impact to occur. It would therefore be valuable to monitor pupil 
outcomes over time. 


The process evaluation collated the views of class teachers, Parent Academy tutors, and parents. 
Parents reported that they had a better understanding of the school curriculum and other school issues, 
which gave them more confidence to engage with their children’s learning. Teachers felt parents who 
attended the Parent Academy became better at communicating with teachers about their children’s 
schooling. Parents were more actively helping children with their English and maths homework. 
Participating parents who had been considered ‘disengaged’ were seen to be more engaged with their 
child’s education after taking part. The full-time area programme manager appears to have been an 
important factor in ensuring that the Parent Academy was developed and delivered successfully. 
Schools generally also allocated a significant amount of staff time to engaging with parents and to 
supporting the intervention. Manualisation of the intervention would help to standardise the Parent 
Academy approach and support consistency in delivery across local areas and over time. 


How much does it cost? 


Total intervention costs comprise one-off set-up costs including tutor training and advertising, and 
ongoing running costs consisting mainly of the material and staff costs of running Parent Academy 
classes and, for the incentivised group, incentives. 


Over three years, the average annual cost of running the Parent Academy would be £641/pupil for the 
incentivised group and £280/pupil for the unincentivised group. 


Table 1: Summary of estimated impacts of the Parent Academy 


Effect size Estimated Security Cost 

(95% confidence interval) months’ progress rating ue 

ar ae ree 0 4888 EEE 
eee ae 0 48288 EEE 
sei eas ie oie -11 &488 ELE 
ea ere 1 &aa86 ELE 


1 Since this report was published, the conversion from effect size into months of additional progress has been 
slightly revised. If these results were reported using the new conversion, all measures would be reported as 0 
months of additional progress. See here for more details. 
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Introduction 


The intervention 


The Parent Academy intervention was designed by the University of Chicago and aimed to provide 
parents with guidance to supplement their child’s learning at home through parent education classes 
comprising two topics: English and mathematics. It focused on the parents/carers of children in Key 
Stage 2 (English primary school Years 3 to 6). The Parent Academy reflected the primary school 
curriculum with the aim of improving pupil attainment. 


To be successful and achieve the desired pupil level outcomes, the intervention had to successfully 
recruit parents to attend Parent Academy sessions. The hypothesis was that a high level of attendance 
and engagement with the intervention content would lead parents to apply their learning to the home 
learning environment and to engage in the appropriate and desired ways with their children’s education. 
This change in parental behaviour would then lead to the expected attainment outcomes for their 
child/children. 


The Parent Academy was delivered from November 2014 to June 2015. In each participating school, 
the parents of Key Stage 2 pupils were randomly allocated to one of three groups: 


1. Invited to attend the Parent Academy; 

2. Invited to attend the Parent Academy and receive a financial incentive of £30 per session; 
and 

3. Not invited to attend the Parenting Academy: the control group. 


Parents in the treatment groups attended fortnightly classes each covering a topic as set out in Figure 
1. A family field trip took place during the school Easter holidays. The provision of sessions was initially 
split by pupil year group and receipt of a financial incentive. The parents of pupils in Years 3 and 4 pupils 
were taught together, as were those with children in Years 5 and 6. In addition, parents who received a 
financial incentive attended separate sessions to those who did not. All parents were offered onsite 
childcare and refreshments. 


Parents in the incentivised treatment group received a £30 incentive for each class they attended. By 
financially rewarding attendance, the developers aimed to increase parental exposure to the learning 
offered in classes and thus increase the effect of the intervention. 


The material presented in the classes was delivered in an interactive way so as to allow parents to learn 
from the tutors and each other. In addition to the classes, parents also undertook homework extension 
activities which enabled them to put into practice the knowledge they gained. These ‘Fun Home’ 
activities were designed to make learning enjoyable and to encourage parents and children to do things 
together and to foster new habits that ultimately enhance the home learning environment. 


The Parent Academy built on a study carried out at the Chicago Heights Early Childhood Center 
(CHECC) which investigated the impact of a similar parent education programme. Findings from the 
CHECC study suggested the programme helped to improve pupil attainment and that incentives offered 
to participating parents were key to its success. Guided by the principles of the CHECC programme, the 
Parent Academy intervention set out to further develop the evidence on improving pupil attainment by 
improving parents’ ability to enhance the home learning environment. However, it is to be noted that 
the CHECC intervention was focused on parents of pre-schoolers in the USA while the Parent Academy 
focused on the parents of primary school children aged 7 to 11 (Years 3 to 6) in England. 


The intervention was not manualised and involved the development of a new programme which was 
delivered in a total of sixteen schools in two English local authorities (eight schools in each local 
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authority). A programme manager (full-time) and an assistant programme manager (part-time) were 
appointed in each local authority to oversee the development and delivery of the programme. With 
ultimate oversight by the University of Chicago, the local authority programme managers’ role involved 
the creation of the curriculum and lesson plans for the Parent Academy, liaison with schools and parents, 
budget management, incentive disbursement, and day-to-day management of the intervention. The 
design specifically included flexibility for the programme managers and tutors to adapt the curriculum 
and course to meet the needs of participants. 


Parent Academy tutors were appointed on a part-time basis to deliver the learning sessions. Four staff 
delivered the sessions in each local authority. These were qualified teachers with knowledge of family 
learning environments in addition to experience of teaching adults with basic skills needs and poor 
English proficiency. In addition to developing the Parent Academy scheme of work and teaching, their 
role included working with school staff to encourage parent retention in the intervention. Parent Academy 
tutors were also required to: 


e attend four formal training events (a total of six days) organised for all tutors. These took place 
every few months throughout the intervention; 

e spend a minimum of two hours preparing and planning for each session; 

e attend a weekly 2-3 hour Parent Academy team meeting (area specific); and 

e be available for parents 30 minutes before and after each session. 


The development and delivery of the intervention were phased. A training session was held for all tutors 
in September 2014 where the English scheme of work was finalised and the first three sessions 
designed. The second meeting was held in December 2014 where the final three English sessions were 
developed. Mathematics sessions were developed in February 2015. A final meeting to reflect on good 
practice was held in June 2015. 


The intervention delivery teams maintained constant communication via email to share resources and 
to make decisions about the content of sessions on an ongoing basis. While all Parent Academy tutors 
followed the same scheme of work and used the same materials and resources, modifications were 
made to ensure the content was relevant to the specific school and to meet individual learning needs. 


In addition, each local authority organised two training sessions for tutors, covering topics outside the 
curriculum: 


Area 1: Safeguarding learners, the Prevent Strategy, Female Genital Mutilation (FGM) 
Awareness; and Promoting Equality and Diversity in the Classroom. 


Area 2: ‘How to make the most of technology’; and using technology and office software 
packages such as iPads, Sharepoint and Office365. 


In total the Parent Academy comprised 13 sessions for parents, 12 taught sessions and one family field 
trip. The first two taught sessions lasted 90 minutes and subsequent sessions were 2 hours long. Fifteen 
parents were expected to attend each session. 
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Figure 1 below provides a summary of the overall structure of intervention design and delivery. 


Figure 1: Intervention development and delivery 


1 day training event 
* Scheme of work for English 
* First three teaching sessions finalised 


Parent Academy starts 
* First 3 English sessions delivered 


2 day training event 

* Reflection on first 3 sessions; 

* Development of final 3 English sessions; 

* Development of Mathematics scheme starts 


Parent Academy 
* Final 3 English sessions delivered 


2 day training event 
* Reflection on English scheme 
* Development of Mathematics sessions 


Parent Academy 
* 6 Mathematics sessions delivered 
* Family event —day trip (Session 13) 


1 day training 
* Reflection on the PA; 


* Sharing of good practice; 
* Contribution to final PA report 


Each session was adapted to curriculum requirements for the Key Stage 2 year groups. Sessions were 
skills-based with a common set of skills underpinning each session: 


e use of questioning to support learning; 

e research; 

e communications, including speaking and listening; 

e building self-esteem of the child and the parent; 

e assertiveness; 

e study skills—resilience, persistence, reflection and self-assessment; 
e use of internet resources to support learning; and 

e role model learning behaviour. 


Each session included a ‘Fun Home Activity’ that parents could complete at home with their child. These 
were designed to make learning enjoyable and encourage parents and children to do things together. 
The topics covered in each session are depicted in Figure 2. 
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Figure 2: Parent Academy session topic coverage 


1. | snodutiontopice | to | snodutiontopice | 


2. Addition, subtraction, number bonds 


at [a mutiniatontacs | [a mutiniatontacs | 


1. . Trip toa museum 


4. | Aoniiontats facts 


5. Fractions, 5 rations, deca percents 5 rations, deca percents cents 


5. Persuasive writing 


6. Planning, drafting, reviewing 


6. Time | 6 rmeandtinetaties timetables 


The control conditions 


The control condition was ‘business as usual’. The Parent Academy did not invite control group parents 
to take part in any alternative sessions, neither were they provided with any Parent Academy or other 
materials. Parents in the control group were also not included in any research activities. This means that 
there is no information about control parents’ circumstances, their perceptions of ‘business as usual’, 
and any relevant support they were accessing. While our research gave no indication of this, it is 
possible that individual schools offered parents in the control group support to improve the home 
learning environment as part of other interventions within the school. The teachers and tutors who 
participated in the research did not have information on parent support activities that control parents 
may have engaged with. 


Background evidence 


Parent education programmes are multiple and diverse, covering a wide range of formats and a variety 
of intended outcomes. While this diversity exists, a vast majority of parent education programmes have 
been directed at families and single mothers with very young children (with some programmes beginning 
in the prenatal months) and have consisted of regular home visits. Studies focusing on these types of 
programmes, such as Parents as Teachers and Nurse-Family Partnership, have found positive effects 
on parenting behaviours (for example, attitudes toward parenting) and child outcomes (Such as cognitive 
skills) (Wagner and Clayton, 1999; Wagner et al., 2002; Olds, 2006). However, inconsistency in results 
across geographic locations as well as variation in outcome measures has made it difficult to draw any 
definitive conclusions about the effects of these programmes. 


Group training programmes that work with parents later in their children’s lives (but still in early 
childhood) tend to show more consistent effects. A notable number of studies, for instance, have 
demonstrated the positive effects interventions such as the Incredible Years Program and Positive 
Parent Program have had on children’s behaviour (Sanders, Markie-Dadds et al., 2000; Sanders, 
Montgomery et al., 2000; Reid et al., 2001; Reid et al., 2004). However, other similar interventions (St. 
Pierre and Layzer, 1999) have proven to be less effective, suggesting that the effectiveness of these 
interventions depends more on the specific content than on the age of the children or the format of the 
programme. 


Research also suggests interventions that target specific groups tend to be effective. A programme 
targeted at adolescent mothers, for example, had significant positive effects on children’s cognitive 
development and their play behaviour (Fewell and Wheeden, 1998), while a meta-analysis of 
programmes targeted at ethnic minorities showed similar effects (Jeynes, 2003, 2007). 


There is a paucity of research on parenting interventions that seek to directly affect academic outcomes. 
While a number of non-experimental studies show a correlation between parents’ involvement in their 
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children’s education and their children’s academic performance (Fehrmann et al., 1987; Griffith, 1996; 
Marcon, 1999) very few parent education programmes aimed at improving children attainment have 
been evaluated through a randomised, controlled experiment. One notable project is Chicago Heights 
Early Childhood Centers’ (CHECC) programme for parents of pre-school age children. The emerging 
findings from this field experiment suggest that programme participation helped improved pupil 
outcomes (Fryer et al., 2015). 


The studies outlined above bring into focus the potential effectiveness and importance of parent 
education programmes. However, many of these studies were conducted in inner city schools in the 
USA. Hence the delivery, educational, and demographic contexts are different. Overall, existing 
research is largely inconclusive in terms of what makes an effective parent education programme, 
especially in relation to children’s academic performance. By building on CHECC’s study, the trial of the 
Parent Academy adds to an emerging body of research that aims to part-fill this knowledge gap. 


Evaluation objectives 


The evaluation consisted of an impact study and a process study. The impact evaluation sought to 
establish whether and to what extent the Parent Academy improved outcomes in reading and general 
mathematics of the children whose parents were invited to take part in the Academy. The impact 
evaluation tested the following hypotheses: 


e Offering participation in the Parent Academy to parents of children in Key Stage 2 will improve 
attainment in reading and general mathematics of these children within a year. 
e Providing parents a financial incentive to attend will increase the effect. 


The process study gathered the views and experiences of those delivering and receiving the intervention 
to assess how the Parent Academy was delivered and its perceived benefits. 


Project team 
The team responsible for delivering the Parent Academy consisted of: 


Edie Dobrez (University of Chicago)—Overall project manager 
Programme manager in each area 
Parent Academy tutors 


NatCen Social Research were the independent evaluators. The research team consisted of: 


Nico Jabin, Research Director 

Sarah Haywood, Senior Researcher 

Jonathan Paylor, Researcher (now at Kings College, University of London) 
Fatima Husain, Research Director 

Adetayo Kasim, Senior Research Statistician, Durham University 


Ethical review 


Ethical approval was obtained by the Chicago University project team for the Parent Academy 
intervention. 


NatCen Social Research obtained ethical approval from its internal Research Ethics Committee for the 
evaluation comprising the process evaluation, pupil testing, data linkage, and analysis of test results. 
Approval was sought for the opt-out process, communications and interviews with parents, tutors, and 
school staff as well as the pupil testing, and data linkage. Approval was granted in September 2014. 
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Trial registration 


This trial was retrospectively registered on 30 September 2015 on the international standard randomised 
controlled trial number (ISRCTN) registry at http://www.isrctn.com/ISRCTN42518213. 
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Methods 


Trial design 


The evaluation was designed as a three-arm randomised controlled efficacy trial, randomising individual 
pupils (except for siblings, see ‘Randomisation’ section, below) into: 


1. Accontrol group, in which parents were not invited to receive any support; 

2. An ‘unincentivised’ treatment group, in which parents were invited to take part in the Parent 
Academy, but not given any financial incentive to attend; and 

3. An ‘incentivised’ treatment group, in which parents were financially rewarded for each session 
they attended. 


An individual-level randomisation design was chosen because the risk of spillover from one family to 
another was judged to be small, making this design the most powerful to detect any possible effects 
(Torgerson, 2001). 


The developers’ previous experience suggested that financially incentivising parents would increase 
attendance and thus improve outcomes. As a result, two treatment arms were implemented to test this 
hypothesis. 


The trial was designed, conducted and reported to CONSORT standards (Schulz et al., 2010). 
Outcome measures 


The primary outcomes are the age standardised scores for Reading and General Mathematics of the 
Interactive Computerised Assessment System (InCAS) (Centre for Evaluation and Monitoring, 2015). 
Age standardised scores are standardised for each age group with a mean of 100 and a standard 
deviation of 15. This score was used because the data from several year groups was combined for the 
analysis. 


Tests were completed interactively by pupils using a computer and head phones. The InCAS Reading 
test is a composite of three test components: word recognition, word decoding, and comprehension. 
The General Mathematics test consists of four components that measure informal arithmetic; formal 
arithmetic; measures, shapes and space; and data handling. 


INCAS was chosen because the wide age range of participating pupils required a test applicable to 
pupils across four school years. InCAS tests accomplish this by interactively adapting the test difficulty 
to the performance of the pupil. This then avoids floor and ceiling effects that would be unavoidable 
using a more targeted non-adaptive test. In order to reduce variability in scores due to age, analysis 
was performed on the age standardised rather than raw scores. 


The tests were administered by NatCen interviewers who were blind to the treatment assignment of the 
children tested. In addition, the test administrators were instructed not to seek out the treatment status 
of individuals by discussing the intervention with teachers or pupils. 


Parent Academy sessions were delivered from November 2014 to June 2015 and pupil testing was 
conducted immediately after the end of the Parent Academy sessions, during the last two weeks of June 
and the first week of July 2015. The evaluation timetable and the school year (ending in mid-July) did 
not permit testing to be conducted at a later date. This time constraint limited the evaluators’ ability to 
test for longer-term effects of the intervention. 


The ‘Pupil characteristics’ section includes descriptive statistics of the outcomes. 
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Participant selection 


Two local authorities, both urban (one in Greater London), were chosen for the trial. These two local 
authorities had experienced difficulties in engaging parents with their children’s education and were 
actively trying to increase parent engagement and motivation to improve the home learning environment. 
The Parent Academy was seen as a vehicle to develop work with parents in the local area. Some schools 
in one area were already trying to find new ways to engage parents in their children’s education. 
Following recruitment of these schools, the University of Chicago obtained a list of all primary schools 
across the two local authorities and emails were sent out inviting schools to join the trial. 


The parents of all pupils in Years 3 to 6 in recruited schools were eligible to take part in the Parent 
Academy. 


Opt-out consent was sought from all parents before randomisation. Information about the Parent 
Academy was provided, pupil data that would be accessed was listed, and NPD data linkage and 
archiving explained. Parents had the option to i) opt out of the trial and ii) opt out of data linkage and 
archiving, even if they chose to participate in the Parent Academy. A copy of the consent form is provided 
in Appendix A. 


Sample size 


The study aimed to recruit 3,000 pupils from 18 schools in two local authorities, assigned to two 
treatment groups and a control group. Due to resource limitations, 500 pupils were due to be assigned 
to the incentivised treatment arm, 600 to the unincentivised, and 1,900 to the control group. 


The initial calculation of the minimum detectable effect size (MDES) assumed 5% attrition and the use 
of Key Stage 1 reading and mathematics outcomes as baseline measures. The baseline was assumed 
to explain between 20% and 60% of variation in the outcome. 


Due to the different sizes of the two treatment arms, the MDES reported is based on the comparison of 
the smaller incentivised treatment with the control group (assuming unequal allocation). For the 
comparison of this group with the control, the MDES was calculated to lie between 0.13 and 0.09 
standard deviations, depending on the assumed predictive power of the baseline measure. The 
minimum detectable effect size for this comparison without a baseline measure was 0.14 standard 
deviations. 


The minimum detectable effect size for the comparison between the two treatment groups was 
estimated at 0.17 standard deviations without the use of a baseline measure. This dropped to 0.11 
standard deviations with the most optimistic assumption about the predictive power of the baseline 
measure (60%, as above). 


All estimates were calculated assuming 80% power and a two-sided statistical significance of 5%. 


The final recruited sample consisted of 2,690 pupils in four years across 16 schools. Of these, 2,153 
were allocated to treatment or control and 1,964 included in the analysis. More detail is given in the 
‘Participants’ section below. 


Randomisation 


NatCen independently carried out the randomisation designed by the project team at the University of 
Chicago and agreed by the Education Endowment Foundation (EEF). It combines stratification, 
matching, and re-randomisation, in order to maximise balance of a range of covariates across the three 
trial groups and reduce heterogeneity among groups during the analysis. 
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Stratification is a frequently used device to enforce balance on a small number of categorical prognostic 
variables. It works by allocating an equal number of participants with a particular combination of 
observable characteristics to each treatment arm. This prevents undesirably unbalanced assignments 
on those characteristics and, when stratification is accounted for in the analysis, ensures greater 
homogeneity among participants. 


Matching is an extension of stratification where a number of participants (one for each trial arm, in this 
case three) are matched to each other based on some prognostic criteria (for example, a baseline score) 
and then assigned at random within the pairs or triplets. Similar to stratification, matching restricts 
assignment to allocations that are considered balanced, and reduces heterogeneity among assignment 
groups at analysis stage. 


Re-randomisation, finally, is another way to restrict the frame of possible treatment assignments to 
assignments with sufficient balance across a range of known covariates. In this way, it is similar to 
stratification, but it allows for the consideration of a greater number of covariates. Re-randomisation can 
be carried out either by pre-specifying an acceptance criterion on a balance check (Such as an 
acceptable p-value on a logistic regression test for the predictability of treatment assignment using the 
covariates), or by using the best assignment? out of a pre-specified number of randomisations (where 
this number is smaller than the number of all possible assignments) (Bruhn and McKenzie, 2009, p.12ff). 


In this trial, following the design set out, all three approaches were used. The practical implementation 
was carried out independently by the evaluation team using an anonymised dataset provided by the 
participating local authorities. 


The implementation aimed to address four requirements: 


1. Similar pupils were allocated (matched) to triplets, and pupils within each triplet randomly 
assigned to the two treatments and the control. 

2. Ineach school only a limited number of randomly selected pupils were assigned to the treatment 

groups, subject to the schools’ capacity constraints; the remainder were allocated to the control 

group. 

Only one of any group of siblings within the four years was allocated to a triplet. 

4. Itwas ensured that pupils in the three treatment groups were as similar to each other as possible 
across a range of variables, while still allowing for random assignment. 


w 


The randomisation thus used a six-step process: 


1. From each family with more than one eligible sibling, one sibling was selected at random for 
assignment. If there existed a non-selected sibling in the same year grouping (either Year 3 & 
4 or Year 5 & 6), the non-selected sibling was given the same assignment (but unmatched— 
see below) as the selected sibling, otherwise the non-selected sibling was excluded. Year 
groupings were relevant because the original intent was for parents of children in school years 
within a group (3 and 4 or 5 and 6) to be taught together, so that a parent of two children in the 
same year group could be considered trained for both children, but a parent of two children in 
different year groups could not. 

2. Pupils were stratified by school, year, gender, English as an additional language (EAL), ethnicity 
(White British versus Other), and whether or not additional data linkage consent was given. 

3. Within each stratum, pupils were ordered by Key Stage 1 Average Point Score (KS1 APS-RWM, 
comprising of reading, writing, and maths scores), and then matched consecutively with two 


? The best assignment is the one resulting in the highest p-value when modelling the relationship between the 


chosen covariates and the treatment assignment. 
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other pupils to form a triplet. If the number of pupils in a stratum was not divisible by 3, one or 
two pupils were randomly removed to create adequately sized strata. 

4. To deal with the fact that the stratification in step 2, due to the many stratification factors used, 
created many groups with only one or two pupils, pupils removed at step 3 were pooled; 
stratified by school, year and whether or not additional data linkage consent was given; and 
matched to triplets as described in step 3. If the number of pupils in a stratum was not divisible 
by 3, one or two pupils were randomly removed to create adequately sized strata. The removed 
pupils were excluded. All triplets formed in steps 3 and 4 were pooled. 

5. By school, triplets were randomly assigned to one of three triplet assignment options (T1-T2-C; 
T2-T2-C; and C-C-C), reflecting the resource availability in each school for offering treatments 
1 and 2°. Within each triplet, pupils were then randomly assigned the assignment options 
available. 

6. Steps 1-5 were carried out 1000 times, each time testing balance on the original stratification 
variables (school, year, gender, EAL, ethnicity, data linkage consent) plus five additional 
balance variables, using multinomial logistic regression. The additional balance variables were 
special educational needs status (SEN), eligibility for free school meals (FSM), term of birth, 
banding of the school’s location on the Index of Multiple Deprivation 2010 (IMD) and pupil 
attendance rate. The randomisation with the highest p-value was used for assignment. 


The randomisation was implemented by NatCen using Stata 12 (StataCorp, 2011). 
Analysis 


In general, analysis of randomised controlled trials should take account of the method of randomisation. 
Ignoring the randomisation method still produces the correct mean estimate of the effect but results in 
unnecessarily large standard errors, and thus unnecessarily wide confidence intervals. However, in the 
case of this study, we believe that the randomisation method was so complex that taking all aspects of 
randomisation into account risked introducing error. As a result, we conducted a more straightforward 
analysis as primary analysis, accepting wider confidence intervals as the more robust approach. We 
considered other aspects of randomisation as secondary sensitivity analysis. 


The primary data analysis compared each intervention arm with the control group using a multilevel 
model to account for heterogeneity between schools and for differential treatment effects by school. 
This model adds to the classical multilevel model which only assumes school as random effect if the 
intervention effect is constant across schools. We presented an unadjusted model which estimated the 
effect of the treatment considering only the relevant baseline Key Stage 1 outcomes (KS1 reading for 
the reading outcome and KS1 maths for the maths outcome). The main (adjusted) model included 
baseline scores as well as the stratification variables available (school year and gender) and FSM 
eligibility. Due to restrictions encountered in accessing de-anonymised data from the National Pupil 
Database, two stratification variables could not be included in the analysis: EAL and ethnicity. In order 
to allow for comparability with the subgroup analysis of pupils eligible for free school meals, we included 
FSM in the primary analysis. Pupils with missing outcome data were excluded from analysis, but a 
sensitivity analysis looked at the effect of imputing missing outcome data. 


The results of the analysis are presented as effect sizes. Effect sizes are a unit-free and normalised way 
of presenting results that allows for comparison between the effects of different treatments. The 
calculation of effect sizes here followed the methodology for Hedges’ g, using the differences in means 
between treatment and control groups as numerator and the pooled standard deviation based on the 
total variance from a multilevel model as denominator. The effect size calculation was based on Hedges’ 
(2007) approach assuming unequal sample size per school. 


3 Every school offered at least as many T2 spaces as T1 spaces, hence there was no need for T1-T1-C triplets. 
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A number of secondary analyses were carried out. First, as part of sensitivity analyses, a classical 
permutation test to estimate the main model’s p-value was performed. Following Fisher (1935), this 
presumes that the null hypothesis of no treatment effect is true and that therefore the treatment effect 
calculated from any permutation of treatment assignment is equally likely. This then allows the 
estimation of the likelinood of the observed effect if the null hypothesis is true (the p-value) by examining 
the proportion of estimated effects greater than the one actually observed. We estimated the p-value by 
permuting treatment allocation 1,000 times to generate a distribution of effect size under the null 
hypothesis. This permutation test was based on simple randomisation without replacement and without 
any further acceptance criteria. 


Second, we analysed results taking account of the re-randomisation used in the original allocation 
process. The principle of this analysis was similar to that of the permutation test described, but the 
random allocations were subject to the same acceptance criteria as the original randomisation. 


The re-randomisation analysis should therefore have created 1,000 random assignments of which each 
was the most balanced of a run of 1,000 possible assignments. However, randomly selecting siblings 
for matching in each round of randomisation as well as the inclusion of pupils without data linkage 
consent (see ‘Participants’ section below) caused a change in the identity of the pupils actually assigned 
to matched triplets and a fluctuation in the number of matched pupils assigned to each treatment. This 
was problematic in terms of re-randomisation, as each new assignment instance was no longer a 
reshuffle of treatment assignments. The assumptions underlying the permutation test were thus violated 
and the re-randomisation test could no longer be run. 


One way to approach this problem at the re-randomisation stage would have been to restrict the set of 
valid randomisations to instances including only the pupils included in the original final assignment. 
However, due to the use of matching based on a continuous variable, holding all required parameters 
fixed during the re-randomisation would have resulted in re-assigning pupils merely within their triplets. 
As aresult, at the analysis stage, we altered the acceptance criteria for a valid random assignment from 
‘most balanced allocation in 1,000 runs’ to ‘achieving a minimum balance in the group assignment’. This 
minimum, measured as p-value from a logistic regression predicting group assignments, was set at 
p=0.5. 


The 2,153 children originally assigned to matched triplets were thus re-randomised to generate a matrix 
of 1,000 acceptable random assignments according to the new acceptance criterion. 


Under the null model of no treatment effects, post-test scores are expected to be independent of re- 
randomisation runs (Morgan and Rubin, 2012, p.3). Using the pre-defined re-randomisation matrix, we 
performed: 


e simple unadjusted analysis between post-test scores and the intervention groups. We reported 
quantile-based 95% confidence intervals for the average intervention effects across the re- 
randomisation runs. The proportion of re-randomisation results greater than the actual “trial 
intervention groups” was reported as p-value; and 

e adjusted analysis for the stratification and balance variables using a multilevel model. 95% 
confidence intervals and p-value were calculated as described above. 


Third, we conducted a sensitivity analysis for missing data by reporting the distribution of missingness 
by intervention groups and the baseline factors. A logistic regression model was used to investigate 
whether the missing data were independent of the stratification variables and intervention groups. We 
also performed multiple imputations to assess the impact of missing data by comparing the results with 
and without imputations. The multiple analysis was based on 50 imputations. Each of the 50 imputed 
data was analysed separately and the results were converted into a single result based on weighted 
average accounting for both within and between imputation variation. 
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Fourth, we assessed the impact of the interventions on pupils eligible for free school meals by carrying 
out the primary analysis using only FSM eligible pupils. 


Finally, we explored the influence of parental attendance at parenting classes. This type of analysis 
allows the estimation of a complier average causal effect (CACE). Our analysis estimated the 
intervention effect taking account of the dose of treatment received by participants (measured as 
proportion of possible sessions attended by each invited parent). The estimate was biased as the factors 
inducing parents not to attend were also likely linked to the measured outcome, and non-attending 
parents were effectively removed from the treatment but not from the control group. Apart from the 
addition of the attendance measure, the estimation used the same model as the primary analysis. 


All analyses were conducted on an intention-to-treat basis. Analyses were restricted to pupils matched 
to triplets, plus any siblings in the same year grouping (who were automatically assigned to the same 
treatment condition as set out in the ‘Randomisation’ section). Pupils that could not be matched to triplets 
were excluded from the analysis. This affected cases: 


e with missing stratification or matching data; 

e that were randomly excluded to allow for the formation of full triplets; and 

e where siblings were not selected for a triplet and were in a different year group to the selected 
sibling. 


The primary analysis and all multi-level modelling was carried out in R (R Core Team, 2015) using the 
Ime4 package (Bates et al., 2015). All randomisations and re-randomisations were carried out using 
Stata 12 (StataCorp, 2011). Multiple imputation of missing data was carried out in SAS software, version 
9.2 of the SAS System for Windows (SAS Institute Inc., 2010). 


The analysis plan is attached in Appendix B. 
Implementation and process evaluation 


The process evaluation was designed to explore how parents, Parent Academy tutors, and staff in 
participating schools viewed the delivery and implementation of the Parent Academy, and what they 
perceived the impact to be. The process evaluation also investigated the extent of fidelity to the 
intervention and collected cost information from the delivery team. 


A longitudinal approach was taken involving two stages of interviews. This approach aimed to capture 
the views of parents and tutors at the start of the programme and once the programme was almost 
complete. This meant that there were two main fieldwork periods; stage one took place between 
December and February 2015 and stage two took place between May and July 2015. 


Parents, Parent Academy tutors and school staff were all interviewed over the phone by NatCen 
researchers. 


Parents 


Only parents in the treatment group (incentivised and non-incentivised) were invited to take part in the 
research. Parent Academy managers and tutors supported recruitment by distributing research 
information materials at teaching sessions. From the parents who agreed to take part in the research 
18 were sampled for the first stage of fieldwork based on a range of demographic characteristics. These 
included: 


e locality (Area 1 or Area 2); 
e —child’s school year; 
e whether the parent was receiving an incentive for attending the Parent Academy; 
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e the relationship to the child (parent/carer/grandparent/guardian); and 
e the relationship status of the participating adult. 


At the second stage of fieldwork, fewer (12) parents were interviewed. A lower number of interviews 
were planned for the second stage of fieldwork in order to take into account the attrition of parents 
between the start and end of the programme. As these were follow-up interviews, parents who 
participated in the first stage of the process study were invited to take part in a second interview, whether 
or not they had been attending the sessions. All those who took part in a second interview reported they 
had attended at least half the sessions. 


Parent Academy tutors and school staff 
All Parent Academy tutors were invited to take part in the research and at both stages of the research. 


School staff were interviewed during the second stage of fieldwork. Initially, headteachers were invited 
to take part in the interview. If they felt they were not the most suitable participant, they were asked to 
nominate a staff member who had been most involved in the logistics and day-to-day running of the 
Parent Academy in their school. Two headteachers, one deputy headteacher, two parent support 
advisors and one family support worker took part in the research. 


School staff from participating schools were interviewed in order to understand how the Parent Academy 
fitted into school life and what school staff themselves perceived the impact, if any, to be for pupils and 
parents. 


Conduct of interviews 


The content of each interview was based on a topic guide to ensure systematic coverage of key issues 
that addressed the process evaluation research objectives. It was intended to be flexible and interactive, 
allowing issues of relevance to be covered through detailed follow-up questioning. Separate topic 
guides were produced for incentivised and unincentivised parents. 


To minimise the burden on participants, all interviews were conducted over the telephone. Interviews 
lasted approximately 45 minutes. The interviews were digitally recorded and then analysed using 
Framework, a systematic approach to qualitative data management developed by NatCen Social 
Research and now widely used in social policy research. All participants were assured that everything 
discussed in the interview would remain confidential and would be treated in accordance with the Data 
Protection Act. 


Timeline 


Planning for the intervention began in July 2013 with the recruitment of schools. The intervention was 
delivered during the 2014-2015 school year (September 2014 to July 2015). Evaluation activities in the 
field started in October 2014 and ended with pupil testing in June & July 2015. Table 2 sets out the key 
evaluation milestones. 


Table 2: High level evaluation timetable 


Date Activity 

July 2013 Schools recruited in Area 2 

January 2014 Schools recruited in Area 1 

July 2014 Opt-out parental consent obtained by delivery partners 

October 2014 Opt-out parental consent obtained by NatCen 

October 2014 Randomisation of pupils performed based on data received from local authorities 
November 2014 | Intervention delivery begins 
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June 2015 Intervention delivery finishes 


June 2015 Post-intervention data collection (pupil testing) begins 


September 2015 | NPD outcome data extract obtained 


October 2015 Analysis 


January 2016 Delivery of report to EEF 


Costs 


Information on costs was provided by the delivery team project manager. This included set-up costs, 
and staff and material costs for delivery of the intervention. Costs for each area were calculated 
separately and combined to give a per pupil cost. A list of engagement activities was provided by area 
project managers, however we were not given estimates of time spent by Parent Academy staff and/or 
school staff on these activities. The trial was commissioned before new guidance on the systematic 
collection of cost data was developed by EEF. 
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Impact evaluation 


Participants 


The parents of all pupils in Years 3 to 6 of the 16 recruited schools were invited to take part in the 
intervention. Of those, 2,690 pupils in 2,306 distinct families did not opt out. However, parents of 97 of 
these pupils, in 89 families, opted out of an additional request to consent to data linkage. This consent 
was required to allow linking of pupil outcome data to baseline and covariate data from the National 
Pupil Database (NPD). Children whose parents refused data linkage were not included in the analysis 
but the parents were able to attend the Parent Academy. 


Of the 2,690 originally recruited pupils, 1,895 (70%) and 1,923 (71%) pupils were included in the analysis 
for, respectively, reading and maths outcomes. The full detail on attrition is set out in Figure 3. 


It is important to note that the randomisation was carried out using data provided in anonymised form 
by the two local authorities where the Parent Academy was delivered. This meant that when the same 
information was obtained from the NPD for the analysis, some differences existed: 


1. First, data on English as additional language, ethnicity and special educational needs status 
could not be obtained from the NPD, as sensitive data requires parental opt-in consent, whereas 
an opt-out process was used for this study. These variables were thus included only in the 
randomisation and not in the analysis. 

2. One variable, free school meals (FSM) eligibility was defined differently at randomisation and 
analysis stage. At randomisation, FSM related to pupils’ current status, whereas the NPD 
variable records whether pupils had ever been in receipt of free school meals. 


Due to these small differences between the pupil data provided by local authorities and that obtained 
from the NPD, the balance of covariates achieved across treatment groups according to the school- 
based baseline data was not synonymous with the balance achieved according to the NPD data. This 
is discussed in the section ‘Pupil characteristics’ below. 
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Figure 3: Pupil attrition 


Approached (pupils n=2690) 


Declined to allow data 
linkage (pupil n=97) 


Assessed for _ eligibility Excluded (pupil n=460): 
(pupil n=2593) Not meeting inclusion 
criteria (pupil n=0) 
Missing stratification or 
balance data (pupil 
n=146) 

Siblings in different year 
group (n=238) 

Not matchable within 
strata (pupil n = 56) 


Randomised 
(pupil n=2153) 


Allocated to Allocated to 
incentivised unincentivised 


Allocated to control 
(pupil n=905) 
treatment (pupil treatment (pupil 
n=569) n=679) 


Lost to follow-up Lost to follow-up Lost to follow-up 
(moved schools (moved schools (moved schools 
(n=48); unavailable (n=52); unavailable (n=79); unavailable 
for testing: for testing: for testing: 

Reading (n=24) Reading (n=22) Reading (n=33) 
Maths (n=12)) Maths (n=16)) Maths (n=23)) 
Post-test data: Post-test data Post-test data 
Reading: 497 Reading: 605 Reading: 793 
Maths: 509 Maths: 611 Maths: 803 
Analysed Analysed Analysed 

Reading: 497 Reading: 605 Reading: 793 
Maths: 509 Maths: 611 Maths: 803 


Table 3 shows the change in the minimum detectable effect size across difference stages of the trial. 
Our calculations at the protocol and randomisation stage were based on the comparison of the smaller 
treatment arm (incentivised) with the control group. The calculations at protocol and randomisation stage 
did not take account of any clustering of pupil outcomes within schools. In our final analysis, however, 
we used a random effects model to estimate the effect of the intervention. This model allowed for school 
effects, expressed in the intra-class correlation coefficient (ICC). In practice, the occurrence of clustering 
increased the minimum detectable effect size, because the reduced variability of outcomes within 
schools reduced the effective sample size. On the other hand, our baseline scores were more predictive 
of the measured outcomes than we had assumed, which reduced the MDES. 
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Table 3: Minimum detectable effect size at different stages, comparing incentivised treatment 
against control 


BS) ro {=} Number of Correlation [exes Blocking / Power | Alpha Minimum 
pupils between stratification detectable 
(incentivised; pre-test & Or pair effect size 


(oxo) ald go))) post-test matching (1 e) =S<)) 


Stratification, 
2,275 ey matching and 
Protocol , 0.0 —0.6 consi- 80% 0.05 | 0.14-0.09 
(475;1,800) re- 
dered ae 
randomisation 
Not Stratification, 


Randomi- 1,500 (578; 00-06 aes matching and 


0, = 
sation 922) - 80% 0.05 0.15 —0.09 


dered ar 
randomisation 
Analysis 1,290 (497; eae ‘ 
Reading 793) 0.66 0.06 Stratification 80% 0.05 0.11 
Bnelye: || ahete ee 0.66 0.06 Stratification 80% 0.05 0.11 


Maths 803) 
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Pupil characteristics 


Table 4 to Table 9 below present the characteristics of pupils after their random assignment to the three 
trial arms. The tables exclude: pupils for whom stratification variables were missing and who were thus 
excluded from assignment (n = 146); pupils who were excluded from analysis because an older or 
younger sibling was selected at random (n = 238); and pupils selected at random within strata in order 
to allow the formation of proper triplets (n = 56). 


The randomisation relied on anonymised statistics provided by local authorities, which were not 
available at the analysis stage. Instead, the analysis used equivalent data extracted from the National 
Pupil Database, requested after the randomisation was completed. Due to restrictions accessing 
sensitive personal data, we were not able to obtain data on ethnicity and English as an additional 
language (EAL). In addition, the measure for free school meal eligibility provided by local authorities 
used eligibility in the year prior to the study, while the data used in the analysis measured whether a 
pupil had ever been eligible. Finally, the randomisation, in the matching, used Key Stage 1 Average 
Point Score for reading, writing and maths, while the analysis used Key Stage 1 reading and maths 
scores as baseline for the reading and maths outcome measure, respectively. Table 4 to Table 7 thus 
presents the variables used in the randomisation and Table 8 and Table 9, for comparison, the data 
used in the analysis. 


Ensuring balance at baseline was an integral part of the randomisation procedure. The logistic 
regression examining the relationship between the group assignment and the stratification and balance 
variables had a p-value of 0.9998, implying a high degree of balance between groups. However, due to 
attrition as well as the differences in data used at randomisation and in the analysis described above, 
analysis variables may not exhibit the same degree of balance at baseline. Table 10 sets out a 
comparison of baseline reading and maths scores between each treatment and the control group. 
Expressed in standard deviations the differences range from 0.04 to 0.07 and are thus very small. None 
of the differences have statistical significance. 


4 Since the parents of treatment-assigned pupils attend the Parent Academy, siblings of assigned pupils were 
considered ‘matched’ if they were in the same year group (Years 3 and 4 or Years 5 and 6), but ‘unmatched’ 
otherwise. Siblings from the same year group as the assigned pupil are thus included in the analysis but siblings 


from a different year group are excluded. The same rule applied to control group pupils. 
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Table 4: Baseline comparison—stratification variables used in the randomisation 


WElar le) (-) Incentivised group Unincentivised group Ofolnlixe) Mel cele e) 
n/N Percentage n/N (missing) | Percent n/N Percentage 
(missing) ' Te) rey) (missing) ' Xo) 
(sd) 
Year 3 aa 27.2 182/679(0) 26.8 254/905 (0) 28.1 
Year 4 ar 24.8 185/679(0) 27.2 ~-251/ 905 (0) 27.7 
Year 5 ei 23.7 157/679(0) 23.1. ~—«:193/ 905 (0) 21.3 
Year 6 es 24.3 155/679(0) 22.8. | 207/905 (0) 22.9 
287 / 569 50.4 49.9 51.4 
Female 339 / 679 (0 465 / 905 (0 
(0) (0.5) ). |. 6.5) ©) | 5) 
253 / 569 44.5 45.9 44,3 
EAL 312 / 679 (0 401 / 905 (0 
(0) (0.5) ©) | 5) () 5) 
White 259 / 569 45.5 44,3 44.4 
British (0) (0.5) SUEIEES | agg) eet ee) (0.5) 
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Table 5: Baseline comparison—categorical balance variables used in the randomisation 


Variable ates al q\Ve-y-lo me] cole) 0) Unincentivised group (ofel alice) Me] cele] e) 
n/N Percentage n/N (missing) | Percent n/N Percentage 
(missing) (sd) rel) (missing) Jo) 
(sd) 
116 / 569 20.4 20.0 20.3 
SEN status (0) (0.4) 136 / 679 (0) (0.4) 184 / 905 (0) (0.4) 
FSM 229 / 569 40.2 38.6 38.5 
eligibility (0) (0.5) geet Ore): | (aay | eee (0.5) 
Birth term 177/569 
Spring (0) 31.1 222 | 679 (0) 32.7 290 / 905 (0) 32.0 
Birth term 203 / 569 
Summer (0) 35.7 236 / 679 (0) 34.8 312 / 905 (0) 34.5 
Birth term 189 / 569 
Aatunin (0) 33.2 221 / 679 (0) 32.5 303 / 905 (0) 33.5 
IMD (<=6" 102/565 
decile) (4) 18.1 117 / 679 (0) 17.2 176 / 896 (9) 19.6 
IMD (7" 85 / 565 (4) 15.0 112/679(0) 16.5 | 142/896 (9) 15.8 
decile) . . : 
IMD (8" 167 / 565 
decile) (4) 29.6 184 / 679 (0) 27.1 245 / 896 (9) 27.3 
IMD (9% 211/565 
decile) (4) 37.3 266 / 679 (0) 39.2 333 / 896 (9) 37.2 


Table 6: Baseline comparison—continuous balance variables used in the randomisation 


WEVar-\e) (= Incentivised group Unincentivised group lOfol alice) Mel geley e) 


N (missing) | Mean (sd) 


N Mean (sd) N (missing) Mean 
(missing) ' o)) 


Attendance 


(mean 


percentage) 


567 (2) 


95.1 
(4.8) 


678 (1) 


95.1 
(4.3) 


899 (6) 


95.1 
(4.5) 
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Table 7: Baseline comparison — matching variable used in the randomisation 


Variable 


KS1 Average 
Point Score, 


Reading 
Writing 
Maths 


Incentivised group 


N 
(missing) 


569 (0) 


Mean (sd) 


14.6 
(3.5) 


Unincentivised group 


N (missing) 


679 (0) 


Mean 
(sd) 


15.0 
(3.3) 


Control group 


N (missing) 


905 (0) 


Mean (sd) 


14.8 
(3.2) 


Table 8: Baseline comparison — categorical NPD variables from the used in the analysis 


Variable Incentivised group Unincentivised group (ofey alice) Me] cele] e) 
n/N Percentage n/N (missing) | Percent n/N Percentage 
(ralesyiale)) ' Te) FYe[=) (missing) ' Je) 
Ese) 

285 / 564 50.5 50.1 51.6 

Female 339 / 677 (2 465 / 901 (4 
(5) (0.5) (2) 5) (4) 05) 

FSM 320 / 558 57.3 54.9 481 / 894 53.8 
eligibility (11) (0.5) Ste OUI) | (0c) (11) (0.5) 
IMD (<=6 111 / 558 191/894 
decile) (11) 19.9 129 / 677 (2) 19.1 (11) 21.4 
IMD (7% 84 / 558 140 / 894 
decile) (11) 15.1 113 / 677 (2) 16.7 (11) 15.7 
IMD (8 124/558 | 22.2 142 / 677 (2) 21.0 196 / 894 21.9 
decile) (11) (11) 
IMD (9% 239/558 | 42.8 293/677 (2) | 43.3 367 / 894 41.1 
decile) (11) (11) 
Birth term 175/564 | 31.0 221/677 (2) 32.6 290 / 901 (4) 32.2 
Spring (5) 
Birth term 202/564 | 35.8 236/677 (2) 34.9 310 / 901 (4) 34.4 
Summer (5) 
Birth term 187/564 | 33.2 220/677 (2) 32.5 301 / 901 (4) 33.4 
Autumn (5) 
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Table 9: Baseline comparison—continuous NPD variables used in the analysis 


WElar le) (-) Incentivised group Unincentivised group Ofolnlige)] Mel cele] e) 
N Mean (sd) WN ial) Mean N (missing) | Mean (sd) 
(missing) (sd) 

KS1 
Reading 564 (5) aa 677 (2) aa 901 (5) ee 
score ; ; ; 
KS1 Maths 14.9 15.3 15.1 
score eee) (3.6) me) (3.6) s0n te) (3.4) 
Attendance | 564 (5) ee 677 (2) ie 900 (6) as 


Table 10: Balance of NPD data at baseline 


Variable Takers} alehviksx=10 InVAcHan Oe) a) dco) | Unincentivised vs. Control 


WW CexclaMmellncslaclalers Effect size Cercle Mme llncslaclarers Effect size 


KS1 Reading 
Ee ce cc a 


Schools 


Local authorities and the participating schools were recruited on a convenience basis, utilising the fact 
that the two participating local authorities had a pre-existing interest in developing parent-school 
relationships. This can limit the applicability of the findings from this trial to other geographical areas, 
since the local interest may stem from an acknowledged problem with parent-school relationships 
(comparatively low baseline), or, reversely, because of successful previous work with parents and 
schools (comparatively high baseline). However, this type of recruitment is appropriate for the efficacy- 
focused nature of the trial. 
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Outcomes and analysis 
Primary analysis 


Table 11 shows the means of the age standardised reading and maths outcomes of pupils in the two 
treatment and the control groups. Pupils in the control group had a mean score of 99.28 in the General 
Mathematics test and 102.60 in the Reading test; pupils in the unincentivised treatment group had an 
average General Mathematics score of 99.61 and a Reading score of 103.07, while pupils in the 
incentivised treatment group had an average General Mathematics score of 99.11 and Reading score 
of 102.06. 


Table 11: Age standardised test scores 


Takex=ta1ehVitx-\o Me] cole) om Ola) alex-1al envi (y-10 Me] cole] o) Control group 


General Mathematics post-test scores, age standardised 


Number of pupils 509 611 803 
(missing) (69) (79) (119) 
acai sana 99.11 99.61 99.28 

: (97.4, 100.78) (98.2, 101.02) (98.09, 100.47) 
interval) 


Reading post-test scores 


Number of pupils 497 605 793 
(missing) (81) (85) (129) 
ee: 102.06 103.07 102.60 
(95% confidence 

interval) (100.63, 103.49) (101.89, 104.25) (101.60, 103.60) 


Table 12 presents the results of our analysis. The estimates of our main model for the effects of the 
treatments on maths outcomes are 0.01 standard deviations for the incentivised and -0.04 for the 
unincentivised interventions. For the effect on reading outcomes, the main model’s estimate of the 
incentivised treatment is 0.00 and for the unincentivised treatment 0.02. None of the estimates are 
statistically significantly different from zero. 


For comparison, Table 12 also presents the estimated effect sizes of the unadjusted model, which 
takes account only of baseline KS1 outcomes. The results are in line with the results of the main 
model. The table also presents the main model's confidence intervals around the effect size estimate, 
the p-value and the intra-class correlation for each treatment and outcome. 
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Table 12: Effect sizes using primary analysis 


| Maths outcome Reading outcome 


Incentivised vs. control 


Unadjusted effect 


. 0.00 -0.01 
size 
Effect size (main 0.01 0.00 
model) 
Confidence intervals 
(iriain model) (-0.20, 0.22) (-0.17, 0.17) 
p-value (main model) 0.39 0.48 
Intra-class 
correlation (main 0.06 0.03 
model) 


Unincentivised vs. control 


Unadjusted effect 


; -0.04 0.01 
size 
Effect size (main 0.04 0.02 
model) 
Confidence intervals 
(iriain inodel} (-0.24, 0.16) (-0.15, 0.19) 
p-value (main model) 0.76 0.33 
Intra-class 
correlation (main 0.06 0.03 
model) 


Note: Statistical significance of effect sizes is indicated by *, such that * indicates that the treatment effect is significant at the 10* 
level, ** at the 5% level and *** at the 1% level. 


Direct comparison of the two treatments was not undertaken due to the fact that no statistically 
significant effect for either treatment arm was found. However, the similarity in effect sizes of incentivised 
and unincentivised treatment suggests that the increased attendance resulting from incentivising 
parents to take part does not improve outcomes. 
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Secondary analyses 
1. Sensitivity analysis: p-value calculation 


Figure 4 shows the distribution of effects calculated using a simple permutation of treatment 
assignments. The solid blue bar indicates the effect actually calculated in the primary analysis. In line 
with the reported confidence intervals in Table 12, the p-values from the permutation analysis show that 
for either treatment the null-hypothesis of no treatment effect cannot be rejected for either outcome. 


This can be demonstrated using the example of the effect of the incentivised treatment on maths 
outcomes (top left chart in Figure 4). A hypothetical treatment status (treatment or control) was 
repeatedly and randomly assigned to each pupil during the analysis. In each case, the average 
outcome of the ‘treatment’ and ‘control’ group was estimated and the difference recorded. After 1,000 
random assignments the ‘treatment effects’ of the hypothetical assignments were larger than the one 
actually found in 39.1% of assignments. We thus have no evidence to reject the null hypothesis. 
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Figure 4: Permuted distribution of effect sizes under the null hypothesis 
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2. Re-randomisation 


The results of the re-randomisation analysis support the findings from the main analysis. The likelihood 
of obtaining the maths and reading outcomes measured assuming the intervention has no effect (the p- 
value) is 0.44 and 0.49 for the incentivised Parent Academy classes, respectively (compared to 0.39 
and 0.48, respectively, in the primary analysis) and 0.72 and 0.31 for the unincentivised classes 
(compared to 0.76 and 0.33 in the primary analysis). The distribution of effect sizes is displayed in Figure 
5. 


As above, the p-value is derived using a repeated and random assignment of a hypothetical treatment 
status to pupils participating in the trial. This time, the random assignment is restricted in a similar way 
that it was restricted in the original randomisation (See the ‘Methodology’ section). The results for the 
maths outcome of the incentivised group vs control group show that in 43.6% of hypothetical random 
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assignments, the calculated effect size was larger than the one actually measured. We thus cannot 
reject the hypothesis that the treatment had no effect. 


Figure 5: Distribution of effect sizes with re-randomisation analysis 
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Loss to follow-up is very low in both the reading and maths outcome as shown in Table 13Table 13, 
as well as the CONSORT diagram. The loss to follow-up rate for maths was 2.3%, 2.6% and 2.8% for 
incentivised, unincentivised and control groups, respectively. The loss to follow-up rate for reading 
was 4.6%, 3.5% and 4.0%, respectively. 


Table 13: Distribution of missing data 


Outcome 


Incentivised 


group 


Unincentivised 


(Ofoy alice) mel celey e) 


Maths 


Reading 


2.3% 
(12/517) 


4.6% 
(24/517) 


group 
2.6% 2.8% 2.6% 

(16/627) (23/820) (51/1,964) 
3.5% 4.0% 4.0% 

(22/627) (33/820) (79/1,964) 


The loss-to-follow-up model did show higher odds of missing data for Year 4 than for the other years for 
the reading outcome. Also, the multiple imputation analysis resulted in a similar effect size as the 
analysis without imputation. There is no evidence based on this sensitivity analysis to suggest that the 
conclusion of the intervention effect would have been different had there been no missing data. 


The results of the model and multiple imputation analysis are presented in Table 14. They show that the 
multiple imputation analysis resulted in a similar effect size as the analysis without imputation. The loss- 
to-follow-up model did show higher odds of missing data for Year 4 than for the other years for the 
reading outcome. However, overall there is no evidence based on this sensitivity analysis to suggest 
that the conclusion of the intervention effect would have been different had there been no missing data. 


Table 14: Logistic regression analysis of dropout r and multiple imputation results 


DY go) oelUi my (oe (=) pm ole @) aa (2)o\0m @4)] 


MLM with Imputation: 
Difference/Slope (95% Cl) 


scereKe ll ale) Maths Reading 
Intercept -3.23 (-5.21; -1.26) | -2.50 (-3.55; -1.46) ae c= : a ; 
ris aa - -0.09 (-0.53; 0.36) 0.12 (-0.24; 0.47) ae — 
ee tivised- 6.12 (-1.23; 0.99) _-0.20 (-0.84; 0.44) cata ot 
Baseline -0.03 (-0.09; 0.03) 0.02 (-0.05; 0.01) a ia ag 
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Year 3-Year 6 


-0.15 (-1.56; 1.26) 


-0.08 (-1.26; 1.00) 


Year 4-Year 6 


0.69 (-0.29; 1.68) 


-1,11 (-1.76; -0.46) 


Year 5-Year 6 


-0.17 (-0.93; 0.59) 


-0.56 (-1.35; 0.24) 


FSM (0-1) 


-0.24 (-1.01; 0.52) 


0.06 (-0.38; 0.49) 


Gender (F-M) 


-0.03 (-0.74; 0.68) 


-0.09 (-0.51; 0.33) 


4. FSM eligibility 


The analysis of effects on only those pupils eligible for FSM showed a similar picture to the main results. 
As shown in Table 15, the estimated effect of the incentivised treatment was -0.02 standard deviations 
for maths, and 0.00 for reading outcomes. The estimate for the effect of the unincentivised treatment 
was -0.01 for maths and 0.00 for reading outcomes. However, none of the effects were statistically 
significantly different from zero. In addition, the differences in point estimates between the outcomes for 
all pupils and FSM pupils are of a negligible size, and an interaction test of the overall intervention with 
an FSM indicator (not presented here) did not result in a statistically significant different effect for FSM 


pupils. 


Table 15: Effect sizes for effect on FSM subgroup 


WENdatomel el cexe)payzy 


Incentivised vs. control 


Effect size 


-0.02 


Reading outcome 


0.00 


Confidence intervals 


Effect size 


(-0.26, 0.22) 


-0.01 


(-0.18, 0.19) 


Unincentivised vs. control 


0.00 


Confidence intervals 


(-0.25, 0.23) 


(-0.18, 0.18) 


5. Attendance 


Attendance at the Parenting Academy was very low. Over four in ten parents (41%) invited to the 
incentivised and nearly three quarters (73%) of parents invited to the unincentivised group did not attend 
any classes (see Table 16). However, even when controlling for attendance, the Parent Academy did 
not show an effect on outcomes (see Table 19). 
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Table 16: Attendance at the Parent Academy 


Incentivised classes Unincentivised Overall (%) 
(€7,)) classes (%) 
0% attendance 41.2 73.37 58.83 
0, 

Fess (hano02 7.93 11.96 10.14 
attendance 

50% or more attendance 28.82 10.85 18.97 
100% attendance 22.05 3.83 12.06 
Total 100 100 100 


The average percentage of classes attended in the incentivised group was 47%, while it was 14% for 
the unincentivised group. As shown in Table 17 17, incentivising attendance increased attendance by a 
statistically significant 33 percentage points. 


Table 17: Difference in attendance between incentivised and unincentivised groups 


Mean proportion 95% 
of classes oxo) aice(aler= 


attended by interval 
parent 


Incentivised group 517 0.47 0.43 — 0.51 


Unincentivised 
group 


627 0.14 0.12 —0.17 


Difference 0.33 0.29 — 0.37 


Examining observable factors associated with parental attendance at the Parent Academy using a zero 
inflated beta model® (Buis, 2012) found that incentivising parents was a strong predictor of attendance 
even when controlling for pupil prior achievement, free school meal eligibility, pupil class attendance 
and local authority area®. The model also showed that pupil class attendance was a strong predictor of 
parental attendance, perhaps indicating parental attitudes to their child’s education. Overall, neither 
FSM eligibility nor pupil prior attainment (measured as Key Stage 1 Average Point Score for reading, 
writing and maths) had any discernible influence on parental attendance (Table 18). 


Table 18: Marginal effects of influences on parental attendance using a zero inflated beta model 


5This approach models the proportion of classes attended by parents, but treats parents that never attended any 
classes differently from those that attended at least some. 
5 Local authorities led the implementation of the trial in their area and thus had an influence on the effectiveness of 


recruitment and parental engagement. 
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Coefficient 95% confidence 
interval 

Incentivised (dummy) 0.15 0.00 (0.11, 0.18) 
KS1 APS (RWM) (prior 
attainment) 0.00 0.30 (0.00, 0.00) 
FSM eligibili 

SMetg Dnty 0.00 0.64 (-0.02, 0.01) 
(dummy) 
Attendance 0.33 0.00 (0.11, 0.55) 
LA (dummy) 0.01 0.19 (-0.01, 0.03) 


Estimates of the complier average causal effect (CACE) are presented in Table 19, showing the 
estimated treatment effect taking account of the same covariates as the primary model plus the addition 
of an attendance measure for parents in the treatment group (defined as proportion of possible sessions 
attended). As before, the point estimates are very close to zero and indicate that the treatment did not 
have an effect at the time of measurement. The results are, again, not statistically significant. 


Table 19: Effect sizes from model taking account of attendance 


Maths outcome s<r-Lel lave model cere) tay 


Takexsdalenvicx\e MVAcMm ore) alige)| 


Effect size -0.05 -0.03 


Confidence intervals (-0.26, 0.16) (-0.20, 0.14) 


Unincentivised vs. control 


Effect size -0.06 0.01 
Confidence intervals (-0.27, 0.15) (-0.16, 0.18) 
Cost 


Upfront costs for the intervention include costs for promotional activities (including an event for 
headteachers), staff recruitment, equipment, and training. 


To deliver the 13 sessions of the Parent Academy, the main costs incurred were for staff time and parent 
incentives. Staff appointed to deliver the Parent Academy included a full-time programme manager, and 
an assistant programme manager in each area. In addition, four part-time tutors were appointed in each 
area to plan and deliver the sessions. Tutors committed to six training days (four separate events) and 
weekly meetings, and were required to be available to parents before and after each session. Parents 
were given £30 in vouchers for each session attended, making incentives the largest non-staff costs. 


Other running costs included sessional childcare staff, refreshments, payment for transport to sessions 
(paid to parents in Area 1 only), printing/photocopying, and venue hire (in instances where suitable 
space on school sites was not available). 
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In schools, the headteacher in each school took on certain responsibilities associated with organising 
the intervention. School staff in varying roles worked with Parent Academy staff to support engagement 
activities, and attend meetings to discuss progress. The time and associated cost of school staff 
involvement varied across schools and an estimate was not provided by the programme developers or 
collected from schools. 


Cost per pupil 


The three-year average annual cost per pupil’ including all trial participants recruited into the two treated 
groups independent of attendance status (n=1,248°) is £641 for the incentivised and £280 for the 
unincentivised group. Table 20 sets this out in more detail. It is important to note that set-up costs have 
been divided equally between incentivised and unincentivised groups, although some costs arguably 
would be incurred in full if only one group was run (for example, the event for headteachers). 


Table 20: Total cost per pupil 


Takex:VaneMicx-lo Me) cele] oll (afetols1:)) Unincentivised 
group (n=679) 


[oxey-) a 629) Cost (£) per (Orol-) ml (2) a OX01-) a (29 

pupil per pupil 
Start-up cost 9,865 17 9,865 15 
Running costs per year 361,617 636 186,657 275 
Total cost—first year 371,482 653 196,552 289 
Total cost over three years 1,094,716 1,924 569,836 839 
Average cost/year over three years 364,905 641 189,945 280 


When these figures are disaggregated by local authority, the results are quite different: respectively 
£754 and £366 for incentivised and unincentivised groups in Area 1, and £549 and £207 for Area 2 (see 
Table 21 and Table 22, respectively). The reason for the differences in per pupil costs is due to three 
interacting factors: 


1. Staff costs were lower in Area 2. 
2. The greater recruitment success in Area 2 reduced the impact of some annual fixed costs. 
3. The lower attendance rate in Area 2 reduced expenditure on incentives for the incentivised 


group. 


7Costs were calculated by adding a third of the start-up costs to the annual running costs, to give an estimate of 
average annual costs incurred if the intervention was run for three years. 


8 See section ‘Participants’, where the recruitment process is explained in detail. 
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Table 21: Area 1 total cost and cost per pupil 


Incentivised group (n=256) Unincentivised 
group (n=311) 


Cost (£) oxoy-) a ¢ 39) (orok-) a ¢ 2) am OX 0)-) ml ( 9] 

per pupil per pupil 
Start-up cost 7,265 28 7,265 23 
Running costs per year 190,730 745 111,493 358 


Table 22: Area 2 total cost and cost per pupil 


Incentivised Co] Cole} Mmm Olaliaver-lalanry-cemelcelul ol (ar<loy-)) 
(n=313) 
Cost (£) Cost (£) per Cost (£) Cost (£) per 
pupil pupil 
Start-up cost 2,600 8 2,600 7 
Running costs per year 170,888 546 75,164 204 


The above costs exclude school staff time and cost, therefore an estimate of cost for school staff 
involved in parent engagement is likely to be needed in order to provide a more accurate per pupil cost. 


Attendance data collated by the University of Chicago (refer to Table 16) suggests that incentivisation 


helped to engage and retain parents in the intervention. The cost for incentivising all parents should be 
considered when calculating the costs of delivering similar but different interventions in the future. 
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Process evaluation 


This section synthesises the findings from the process evaluation by bringing together the perspectives 
and experiences of stakeholders who participated in the evaluation research. These include: Parent 
Academy tutors who developed and delivered sessions for parents; teachers who supported delivery of 
the intervention; and parents who attended Parent Academy sessions. It discusses the main issues 
related to implementing the Parent Academy, sets out the key challenges, and identifies areas of 
improvement that may strengthen fidelity and replication. All Parent Academy tutors were invited to take 
part in the research along with six purposively selected school staff. The research was conducted in two 
waves, the first designed to understand the development and early delivery of the intervention, and the 
second to understand how delivery progressed, the challenges encountered, and the perceived benefits 
of participation. Table 23 sets out the number of achieved interviews for each type of participant. 


Table 23: Achieved interviews 


excVad(ed| or-lald Achieved 

interviews 
Wave 1 Parents 18 
PA tutors 7 
Wave 2 Parents 12 
PA tutors 7 
School staff 6 


Implementation 


The discussion on implementation is structured around the key elements that were found to be related 
to successful delivery of the Parent Academy, namely: 


e attractiveness of intervention and school motivation to take part; 
e role of the area programme manager; 

e developing Parent Academy session content; 

e flexibility in content development and teaching techniques; 

e practical delivery arrangement; 

e relationship with schools and the role of school staff; 

e parent engagement and retention; and 

e attendance rates. 


Attractiveness of intervention and motivation to take part 
Encouraging parental engagement in their children’s education was identified by some teachers as one 


of their school’s central aims. As such, the Parent Academy aligned with the existing school ethos. There 
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was a similar understanding among tutors of the need for an intervention that bridged the relationship 
between parents and teachers to support children's education. 


‘Well for me the main goal is to, for us to give parents some skills and strategies to help them 
to help their children's education...’ 


Teachers saw the Parent Academy as another approach to engage parents in their child’s education 
(see ‘Parent engagement’ section below). However, their enthusiasm was tempered by a concern 
among teachers of the ‘inequality’ of incentivising some parents and not others to take part. 


Role of the area programme manager 


The area programme managers played a prominent role in ensuring the smooth running of the Parent 
Academy. They were able to provide the dedicated time and effort required to facilitate the development 
of intervention content as the intervention was delivered. Their role included developing strategies to 
engage and retain parents, school liaison, developing the timetable, and managing Parent Academy 
tutors. Their knowledge and experience of teaching adults and project management and leadership 
within an educational context were believed to be critical to the role. The area managers’ local authority 
base supported access to data, provided office space, and helped to publicise the intervention widely. 


Developing Parent Academy session content 


The content of all sessions was developed based on a core framework and lesson plan agreed with the 
developers. Four formal training sessions were held with all tutors from both areas to develop the 
programme. Tutors felt well supported to develop sessions while delivery of the Parent Academy was 
in progress, and weekly meetings in each area facilitated a collaborative approach. 


Concern was expressed in relation to varying skill levels and experience among the tutors and the 
differences in teaching styles which made it more difficult to develop a consistent teaching approach. 
Although working with the University of Chicago staff was viewed positively, the distance was identified 
as a challenge to effective communication. 


‘...we have a very different view on how to put the curriculum and the course across... and we 
feel it's been ignored time and time again. We've actually stopped commenting on the 
resources, we've just totally ignored them and written our own.’ 


A collaborative working approach was facilitated and led by the area programme manager, who provided 
support to tutors on an ongoing basis, was seen as instrumental to the success of the programme. 
Although the frequency of meetings and collaborative approach had come about naturally, tutors felt 
that collaborative content development was beneficial to the programme as a whole. A suggestion would 
be to formalise a collaborative approach in any documents developed to guide future delivery. 


Flexibility in content development and teaching techniques 


Flexibility in content development was identified as important for the effective delivery of the Parent 
Academy. Ownership of content by tutors adapting the content to suit individual teaching styles and 
parents’ abilities were two aspects of flexibility that were considered to underpin successful delivery. 


Moreover, establishing a relaxed learning environment that encouraged social interaction during 
sessions was believed to be a key component of success. Activities and teaching techniques that 
supported this included paired and group work and the provision of refreshments. Importantly, tutors 
strove to create a learning environment where parents ‘/felt] they can make mistakes’. 


This flexibility to adapt teaching approaches was thought to be important in order to accommodate 
parents with varying skills and abilities. The greatest challenge was believed to be delivering sessions 
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to parents with limited or no proficiency in English and poor mathematics skills. The needs of this group 
of parents were a priority when Parent Academy tutors discussed materials and teaching techniques. 
At the other end of the spectrum were parents with a high level of skill who felt they ‘already knew it all’. 
Therefore, the ongoing challenge was to deliver sessions that appealed to parents across the ability 
spectrum which meant that tutors had to adapt each session based on the types of parents attending. 
While the topics and content were covered consistently across sessions and areas, Parent Academy 
tutors and schools suggested that the content could be developed further to complete curriculum gaps 
and include topics such as ‘parenting skills’. Developing a ‘toolbox’ of materials and techniques that 
Parent Academy tutors can draw on to help engage diverse parents with lesson content was highlighted 
as a possible method for increasing engagement and retention. 


Practical delivery arrangements 


Participating schools were asked to identify rooms within school grounds to deliver the Parent Academy. 
Where this was not possible external venues were used. The settings where the Parent Academy was 
delivered varied with some sessions delivered in well-equipped and adequately sized rooms and other 
instances where the tables and chairs (in a school setting) were designed for children but tutors and 
parents had to make do. Important for effectively delivery was a consideration of: 


e sufficient space to allow paired and group work and for parents to interact; 

e accessible venue to parents—close to school or home but not necessarily within a school 
setting; and 

e access to appropriate IT facilities. 


Multiple sessions were delivered during the week and on weekends to accommodate parents who were 
in employment and to separate incentivised and non-incentivised groups of parents. The original 
intention was to deliver the sessions for Year 3 and 4 parents separately from those for Year 5 and 6 
parents. This approach was altered due to low attendance rates and parents across all years were 
invited to attend any session while maintaining the separation between the incentivised and non- 
incentivised groups. 


Based on early feedback from parents, the session timetable was revised to offer lessons during school 
time (So parents could attend while their child was in school) and during the evening and on weekends 
(for parents with weekday commitments). However, weekend sessions were poorly attended. 


Relationship with schools and the role of school staff 


As delivery of the Parent Academy was organised and managed at local authority level, the perception 
of tutors was that the initial engagement of teachers was challenging because: 


e teachers were perceived to be extremely busy; 
e Parent Academy tutors felt they were intruding on the school; and 
e some school staff felt ‘threatened’ by the research element of the project. 


Allocating sufficient time to engage with and build relationships with school staff, and face-to-face 
meetings arranged at times suitable for teachers, were thought to be important in overcoming these 
challenges. 
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Parent engagement and retention 


The initial information letter sent out by the Unversity of Chicago informing parents about the Parent 
Academy received a mixed response. Those who had ignored the letter or failed to engage with its 
content learned about it by word of mouth from other parents, teachers, and school-based Parent 
Support Advisors. Involvement of school staff to convince parents helped with initial parent engagement. 


The staff member] said that, 'This course would be quite good for you to have a better 
understanding of the way your child learns’...1 thought, oh yeah, it sounds good.’ 


Face-to-face contact with school staff was thought to help parents more than written communication 
(letter and text). However, a letter that stood out was enclosed in a gold envelope and contained what 
was perceived to be useful information about their group allocation and a timetable of sessions. 


A desire to support their child (and in some cases, grandchild) to improve their learning, along with the 
potential to improve their own skills, were motivating factors for parents joining the PA. 


‘[I decided to join the academy] so it could help me, really. I'm more aware of how they’re 
teaching in class. How can! get my kids to improve. You know, to do better in life, maybe better 
in their exams.’ 


Clarity in communicating the aims of the Parent Academy and explaining, in accessible ways, the 
potential benefits to parents would be important to successful engagement. 


Retaining parents in the programme required additional efforts by school staff and Parent Academy 
tutors, who used various approaches (email; text messaging; door knocking; telephone calls; speaking 
to parents when they picked up their child from school) to remind parents of up-coming sessions. Due 
to data protection considerations, school staff were mainly responsible for contacting parents, with face- 
to-face contact perceived to be the most promising method. 


The provision of refreshments was thought to have supported engagement and retention of parents and 
contributed to the relaxed and informal learning environment necessary to engage diverse parents, 
including those who may have had barriers to learning due to their own experiences of schooling. 


Attendance rates and financial incentives 


The offer of £30 per session to attend the Parent Academy motivated some of the parents from the 
incentivised group to continue attending. This group of parents felt it was a ‘fair payment for their time, 
without which attending the Parent Academy would not have been appealing. 


In some cases, the financial value of the incentive provided participants with additional motivation to 
attend sessions more regularly than they would have done otherwise. 


‘My honest opinion was | would've attended but probably didn't make that much effort to attend 
every week ‘cause obviously I've got three children...but with the vouchers | knew | was getting 
£30 vouchers, which is a lot. That was my one week's shopping out of the way for the food—so 
| was attending regular.’ 


Attendance data was collected for each area by the University of Chicago. Table 24 below presents 
attendance data by local area. Table 25 presents attendance rates by treatment group. 
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Table 24: Attendance rates by area 


Area 1 (%) Area 2 (%) Overall (%) 


100% attendance 15 9 11 


50% or more 


attendance 20 AY us 
0, 

Less than 50% 8 2 10 

attendance 

0% attendance 57 63 60 


Base: Pupils assigned to treatment groups (n = 1,248) 
Table 25: Attendance rates by treatment group 


Incentivised classes Unincentivised Overall (%) 


(CZ) classes (%) 


100% attendance 21 4 11 


50% or more 


2 1 1 

attendance 8 ‘ 
0, 

Less than 50% 8 12 10 

attendance 

0% attendance 43 74 60 


Base: Pupils assigned to treatment groups (n = 1,248) 


Between the incentivised and the unincentivised groups, there was an 18 percentage point difference 
between the proportion of parents attending at least half of the classes, and a 17 percentage point 
different in the full attendance figures. The variation in attendance suggests that the £30 voucher given 
to parents after attending each session was successful in increasing attendance rates?. 


Those who attended the Parent Academy regularly found the fortnightly sessions and the 90 minutes 
to2 hours length of sessions to be appropriate and manageable. Poor attendance meant that Parent 
Academy tutors found it difficult to deliver sessions in the intended way. 


° This is based on a final milestone report on the Parent Academy prepared by the University of Chicago team and 


submitted to the Education Endowment Foundation in September 2015. 
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Fidelity 


When assessing fidelity to the programme delivery model there are two aspects to consider. The first 
relates to whether the programme content was developed as intended. The second relates to fidelity in 
how the Parent Academy sessions were delivered across the different sites. 


The Parent Academy was inspired by the Chicago Heights Early Childhood Center (CHECC) parent 
programme. The CHECC programme focuses on parents of pre-school children (ages 3 to 5), while the 
Parent Academy is delivered for the parents of English primary school children in Years 3 to 6 (ages 7 
to 11). Programme developers agreed a core framework—a Scheme of Work—for the Parent Academy 
(based on the English primary school curriculum) based on which lesson plans were developed. This 
development of the programme was executed as intended, however gaps in topic coverage were 
highlighted by Parent Academy tutors. 


Delivery of sessions based on these lessons plans was reported as being consistent across both areas. 
Flexibility was introduced in the use of materials and teaching techniques deployed in order to engage 
parents across a range of skills. 


A number of factors that may risk fidelity were identified: 


e Tutors had insufficient time to build effective relationships with schools—more time was needed 
to work with school staff, and to involve them with the Parent Academy, primarily to support 
parent engagement and participation. 

e Adesignated member of school staff with time ring-fenced to support the Parent Academy was 
considered important to its’ success. In some schools the support of Parent Engagement 
Officers or Family Support Workers was critical to the successful engagement of parents. 

e A lack of suitable space to deliver lessons that involved group-based activities. This was 
identified as an issue in some locations. Availability of a suitable space may also have restricted 
the number of sessions that were delivered in any given week to cover parents of children in 
Years 3 to 6 and to provide separate sessions for non-incentivised and incentivised parents. 
Parents preferred a location that was familiar to them (Such as a school setting) and the use of 
external settings may have affected attendance rates. 

e =A lack of coordination among Parent Academy tutors could threaten consistent delivery. Weekly 
meetings and sessions to develop content in a coordinated way worked well. Even if content is 
manualised for consistent delivery, frequent meetings would need to be scheduled to facilitate 
a collaborative approach. 

e Poor or ineffective communication about the Parent Academy and what it was trying to achieve 
for parents and children alongside a weak understanding of personal constraints (Such as 
disengagement from education or limited time and resources to attend) were identified as risks 
to successfully delivery. Sufficient resources would need to be allocated for face-to-face parent 
engagement activities. 


Crucial to ensuring a consistent delivery approach was the area programme manager. This role helped 
to foster a collaborative approach among Parent Academy tutors and facilitated the relationship with 
schools. Face-to-face communication with school staff helped to secure delivery locations and develop 
a session timetable that suited schools and parents. In addition, flexibility on the part of Parent Academy 
tutors was an important skill so that the session timetable could be amended to suit parents’ availability. 


Perceived outcomes/benefits 


The overall view among school and Parent Academy staff was that there is an ongoing need for 
parenting education programmes that teach parents the skills and knowledge they need to support their 
child’s education. 


Education Endowment Foundation 44 


Parent Academy 


With regard to the specific Parent Academy intervention, perceived benefits included: 


e Improved knowledge of the curriculum—parents were better able to understand educational 
terminology and the structure and content of the educational curriculum. This familiarity was 
believed to have increased parents’ confidence to become more involved in their child’s 
learning. 

e Increased confidence of parents in engaging with their child’s school—the perception was that 
parents were communicating in different, improved ways with teachers during informal catch- 
ups and at parents’ evenings. 


‘1can ask them [child’s teachers] more questions as well, how well she's doing in her 
maths or what she's been taught...’ 


e Increased involvement of parents in their child’s learning—the evidence suggests that parents 
were more actively helping children with their English and maths homework. This was perceived 
to have been partly due to an improvement in parents’ listening and questioning skills when 
interacting with their child. The ‘Fun Home Activity’ which accompanied each session was 
perceived to have helped parents to bond with their child. 


‘Sometimes when walking to school, we'll play games on working out percentages and 
divisions, and mathematics problems, which | didn't do beforehand.’ 


e Improvement in pupils’ English grades—school staff noted an improvement in pupils’ English 
grades which they speculated was due to improvements in parent-child interactions in relation 
to schooling and school work. 


‘The major benefit, | thought, was improvement with my child's education... yesterday | 
had his parents' evening with his classroom teacher... He made improvement, ‘specially 
the English. He exceeded his English grade... He was below the average most of time. 
This time he's exceeded his Year 5 English level.’ 


e Increased parental engagement—the perception was that parents who had previously appeared 
‘disengaged’ were more engaged with their child’s education after having taken part in the 
Parent Academy. These parents were thought to have overcome their discomfort of being in a 
school setting. 


Moreover, a renewed interest among parents to improve their own skills and education was evidenced, 
as was an increased awareness of their role in their child’s education. Parents with English as an 
additional language (EAL) found that their English had improved as they were forced to speak English 
during sessions. Alongside this, children were reported to be pleased that their parents were attending 
the Parent Academy. 


Attending classes with other parents and taking part in group activities (helped by the provision of 
refreshments) meant that parents were interacting more with each other. This was perceived to have 
strengthened relationships and led to the formation of new friendships. This was valued and appreciated 
as an additional benefit of the intervention. 


‘Gosh it was just, it was just good to get out of the house, and you know, and just sit and have 
a cup of tea, and not worry.’ 


There was an acknowledgement among school staff that change among parents and their children is 
likely to be gradual with further benefits to be accrued over time. 
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Formative findings 


The findings from the process study indicate that the intervention in its current form was well received 
by attending parents and schools alike. In both areas there is a commitment to continue to deliver Parent 
Academy courses. Intervention delivery could be strengthened by manualising the topic coverage and 
lesson content or by developing a database of resources. At the same time some flexibility has to be 
encouraged to allow tutors to develop teaching techniques collaboratively in order to deliver content in 
ways that accommodate parents’ skills and abilities. Incentives motivated parents to attend and 
information about this offer should be included in the initial publicity materials to encourage engagement. 
Lastly, the provision of refreshments and childcare, for those who needed it, was believed to facilitate 
initial engagement and ongoing attendance. 


Aspects of intervention delivery that should be reviewed are: 


Communications: a range of approaches to communicating information about the intervention and its 
objectives could be set out during the planning stage. Establishing a personal relationship with schools 
and parents and regular communication with parents was found to be essential to keeping parents 
engaged. 


Promoting the Parent Academy (to encourage retention): ongoing promotion to retain parents in the 
intervention was a modification introduced during the delivery of the Parent Academy. To be successful, 
approaches such as promoting the intervention during parent-school meetings, may need to be agreed 
with schools from the start. 


Topic coverage: parenting skills was a topic which was not covered in the Parent Academy. This was 
identified as an important gap should a review of the content be conducted. The challenge to delivery is 
developing content that engages parents with varying skills and at the same time covers the KS2 
curriculum in sufficient detail to be relevant for each year group. 


Sessions for parents and children: It was believed that delivering a small number of sessions where 
children and parents attend together (perhaps at the start of the Parent Academy and at the end of each 
scheme of work) may encourage parents to take part. 


Modifications suggested by the University of Chicago team 
To be considered in any future development of the project are additional suggestions made by the 
intervention developers. These include: 


e weekly sessions (instead of the fortnightly sessions delivered); 

e each session to last 2 hours; 

e higher level of incentivisation than the £30 shopping vouchers per session given to parents; 

e parents champions to support recruitment and attendance; 

e self-selection of parents to the intervention; and 

e adaption of the content to cover the Key Stage 1 curriculum and to address the transition from 
Key Stage 1 to Key Stage 2 (that is, from Year 2 to Year 3). 


To replicate the area-based delivery approach, local authority involvement by appointing an area 
programme manager is likely to be important. Four Parent Academy tutors per area to deliver sessions 
for the Key Stage 2 curriculum across eight schools was perceived to be sufficient. A school-based 
delivery model may be logistically less complicated but would require dedicated staff time to plan and 
deliver the intervention. 
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Although the Parent Academy was well received by those who attended sessions, the high percentage 
of ‘no-shows’—parents who never attended—means that identifying effective ways to engage and retain 
parents should be a priority for initiatives with similar aims. 


Control group activity 


The trial assumes a ‘business as usual’ approach to parenting involvement in the control group. This 
means that parents of children in the control group were not invited to take part in any Parent Academy 
sessions but individual schools may have run other parent-focused interventions. We are not aware that 
such interventions actually took place, but research with parents in the control group was not included 
in the study. 


An individual-level randomisation design was chosen because the risk of spillover effects if parents 


participating in the Parent Academy shared their learning with non-participating parents was judged to 
be small. However, a measurement of any spillover was not part of the research design. 
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Conclusion 


Key Conclusions 


1. There is no evidence that offering free Parent Academy classes improved mathematics or reading 
outcomes for the children in the trial, even when parents were given a financial incentive to attend. 


2. In general, parental attendance at sessions was very low. However, even when the evaluators 
took this into account they found no evidence that parental attendance improved pupil outcomes. 


3. Offering financial incentives improved attendance at Parent Academy, suggesting this may be an 
effective way to engage and retain parents in interventions of this type. 


4. Ingeneral, evaluation participants felt that attending Parent Academy gave parents the confidence 
and skills to engage more effectively with their children’s learning. 


5. Staff reported that having a designated project lead in each school and using multiple methods to 
engage parents (rather than just written communication) were necessary for successful delivery. 
Schools need to consider these cost implications when deciding whether to adopt this intervention. 


Limitations 
There are four main limitations on the security of findings from this trial: 


e Parental take up of the intervention was very low. 

e Acomplex randomisation process excluded many participating pupils. 
e The intervention was not sufficiently developed. 

e =There was a high level of flexibility built into content development. 


The very low take-up of the intervention makes finding any impact very unlikely. This trial directly and 
unbiasedly assessed the effect of the offer of the Parenting Academy, with or without an incentive to 
take part, rather than the effect of taking part itself. This trial provides less insight into the actual effect 
of training for parents among those who take part in the tuition on the educational outcomes of their 
children. However, the analysis accounting for attendance that we have conducted does not suggest 
that there are strong effects among the groups of children whose parents did attend the Parenting 
Academy. Any future research on this approach could test different ways of engaging parents, including 
focusing on particular subgroups. 


The randomisation approach, due to missing data and the matching requirements, led to the exclusion 
from analysis of some 30% of participating pupils. While all pupils were excluded at random and mean 
effects should therefore not be biased, a simpler randomisation strategy could have had a larger sample 
size and thus a smaller minimum detectable effect size. 


Important aspects of the intervention were developed during the trial. This includes specific lesson plans 
and also activities undertaken to engage parents. As a result, even if take-up had been higher and the 
randomisation had not affected the sample size, it would have been difficult to assess fidelity and to 
determine what exactly had caused any estimated effect since the intervention was reacting to 
circumstances. 


Assessing fidelity was all the more difficult due to the high level of flexibility built into content 
development. Programme managers and tutors felt able to disregard the advice and resources provided 
by the University of Chicago to develop lesson content. 
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There are other limitations, particularly the short follow-up period, the risk of spillover, and external 
validity problems, as discussed below. 


Pupils were followed up only two weeks after the end of the intervention. Given that the Parent Academy 
aims to influence pupils’ results only indirectly, a longer follow-up period would be more appropriate to 
detect any effects. 


Spillover may have occurred if parents participating in the Parent Academy transmitted their knowledge 
or attitude to engaging in learning with their children to other parents. Although we have not measured 
what the control group parents did, we judge the spillover risk to be small. We hypothesised that the 
intervention’s main transmission mechanism is the knowledge parents gain from participating, and that 
the likelihood of parents teaching other parents is low. It is possible, however, that an intervention such 
as this, if it were effective, would work primarily via a change in attitudes rather than learning, which 
could be transmitted to other parents, making spillover more likely. However, we do not explore that 
possibility here as we do not have any insight into the attitudes and views of control group parents. 


Finally, the trial took place in 16 schools in two urban local authorities which means that the scope of 
the intervention (and the trial) was limited. The extent to which schools engaged with the intervention 
varied. Those more committed to parent engagement or with more enthusiastic or involved 
headteachers were more willing to allocate resources to support the intervention. This contextual 
variability, alongside issues with initial engagement and retention of parents and the availability of 
suitable venues, limits the applicability of findings to other areas. 


Interpretation 


The trial indicates that inviting parents to English and maths sessions to improve their children’s home 
learning environments does not affect outcomes of pupils in the short term. This is true for both maths 
and reading outcomes. There is an indication that financial incentives increased parents’ attendance but 
that this did not alter the mean estimate of outcomes. 


Even if an effect had been found, in the first year, the intervention was moderately expensive, at £653 
per pupil per year for the incentivised and £289 for the unincentivised group. Averaged over three years, 
the annual cost would be £641 per pupil for the incentivised and £280 per pupil for the unincentivised 
group. Intervention effects would have to be very large to justify this level of expenditure. 


Evidence from the process study found that parents who attended sessions appreciated the additional 
learning which they saw as benefitting themselves and their children. Approaches to engaging and 
retaining parents were modified during the intervention. As engagement approaches were not the focus 
of the study, it is difficult to determine which strategies were most effective. 


Parent Academy tutors valued the collaborative process undertaken to develop and deliver the 
intervention. Flexibility to adapt teaching techniques to include parents with a range of skills was 
considered an important aspect of the approach undertaken. Even if the content of the Parent Academy 
is manualised, there would be value in fostering a collaborative approach to facilitate the promotion of 
good practice in teaching and to help to build relationships between Parent Academy and school staff. 
The involvement of school staff depended on the level of effort expended to engage and retain parents. 
A designated school contact for the Parent Academy programme manager and for parents was helpful, 
but it was difficult to estimate the time schools would need to set aside due to the range of engagement 
activities deployed. 


The null results of this trial raise the possibility that the intervention logic may contain weaknesses— 
perhaps due to the initial lack of specificity of the Parent Academy programme or its development from 
an intervention intended for the parents of pre-schoolers. It may be that insufficient consideration was 
given to the local cultural context within which parental behaviour and perceptions of this type of 
intervention are likely to be different from those of parents in Chicago Heights. We would also question 
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whether the dosage (the number of sessions delivered) was sufficient to achieve the level of change 
needed to observe strong effects. 


Future research and publications 


This intervention would merit further evaluation once the shortcomings of the intervention have been 
considered and addressed to the extent possible. 


Intervention specificity: The intervention was developed while delivery was taking place, which means 
that context specific modifications may have been made to lesson plans and content. While some 
flexibility in teaching techniques is important to accommodate the range of skills among parents, steps 
to manualise the intervention may help to support consistent delivery across a wider geographical area. 
This includes setting out clear lesson plans aligned with parental and pupil learning requirements, and 
specifying effective ways to engage and retain parents. 


The small number of schools: The small number of schools taking part in this trial may mean that the 
evaluation findings are anomalous, in part as a result of context specific issues relating to the recruitment 
of schools as well as the type of parents who attended (and those who did not). Conducting a larger trial 
should help to minimize the influence of contextual factors. 


Parental involvement: Testing the effectiveness of parent engagement activities would help to better 
understand how attendance rates for this type of parenting intervention can be boosted. 


Evaluation timescales: The evaluation assessed the outcome of the intervention immediately after the 
end of the intervention. Given that parental behaviour change is an intermediate step to improvements 
in attainment outcomes for children, this short follow-up period is likely to limit the detection of any effect 
that the intervention may have. If the evaluation is limited to a short evaluation timetable, or where it 
focuses on an intervention where the target group is different from the group on which impact is being 
measured (such as the Parent Academy), a study of intermediate outcomes would be beneficial. In this 
case, a study of intermediate parent outcomes and behaviour change within the home learning 
environment would provide valuable insight into the effectiveness of the intervention. 
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Appendix A: Parent consent form 


Ln? il 
an Parent Academy NatCen = nent 
mist +i i Social Research that works tor society in 

Dear Parent or Carer, 


RE: Parent Academy Programme 
ee nt In the Parent Academy Programme which is tunded 
he Sedan Creioment eal Endawment Foundation. The Involves. offering some parents the chance to attend =| 
couse to find cut more about wat their chiaren are eaming at school This will alow researchers at 
NabGen Sodal Research and the University of Chicago to look at the effects of the course on children’s 
achievements In English and Maths at school. Please find endosed a leafiet with mone details about the 
Parent Academy Programme, Including the organisations Involved. 


We are waiting bo a5k for your es eee age, gender 


the Fischer F: ‘Trust and In anonymised form to the UK Data Archive. More Information about tis 
pettcbenciecy + ous ‘apa ae . asked questions can be found overleaf. ; 


The data trom this stuny wil be stored securely and freaied win the stictest confidence. Information will 
be-used for research purposes only and no schoots, parents or pupes wil be identified In any reports. 


‘YOUrSs sincenety, 


Hf you are happy with everything In this letter you do not need to do anything. you donot want 
Endowment 


| crore i a lor ela meer SF ep ll 
(Data Archive please fl In the sip below and return fio the school by the 23nd Ochober. 


OPT-OUT SLIP I do net wari my child to 
« Have their data Inibed to the National Pupll Database 
« Have thelr dai stored In ne Education Endowment Foundation Data 
Araive 
TO a a Ss 
IO PN SI a a cn a a et cee eta eos 
TOI ih a 8 
‘Your telephone number (optional)... 


On a i Ue cts 
Please relum this slp to your school by the 23nd Oetober if you wish to opi-owt This will not affect your 
chance of selection to the Parent Academy courses. 
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Frequently Asked Questions: 


@ Whar abour data and confidentiality? 


PUD! test reeponses and other data requested trom the school will be collected and accessed by the 
research teams at NabCen Social Research and the University of Chicago and will be treated In the 
sinciest confidence and in line with the Data Protection Act. Data will be submitted in anonymised farm to 


Vilih the permiseion of the Department for Education, Gaia will be matched wiih the Matonail 
ei pe ge ne meer peel a esicpaborelcree ele 
Education Endowment Foundation’s cantacior, Fiether Family Trust Cate Archive. 


@ Whar is the Nadonal Pupil Database (NPD)? 


The NPD Is a database, marsaged by the Depsiient fx Exiucolon (O8E2. which tis hntamaton al 
individual pupi-level about children's education at different stages, Including test and exam resufs for al 
children in the state sector In England, a5 well as non-maintained special schoois, shah form and Further 


collages and (where Independent schools. The NPD also holds information 
shen Fapies Keen one eee ger acaed medicalliemenl anti erucanrid aaiwnet 


@ How will che data be used? 


The data tom this Tor PUD. Th Urtueredy of Gttess (Chicago and NabCen Socal Research who will 
prepare research The Unk Se ee 

TES POnSes and tata from ihe local aumortiy. NabCen Social Research wil have acess to ail 
Sia No Fai aduad sects, parents or pegite wil be Weriaaed any Pepa areing from iis research. 


Data from each of the schools imvotved in the wil be analysed anonymously, and findings wil be 
reported anonymously, for example “1 3, of had numerical skilis atx level or above”, so it will not 
be possinie io identity individual pupils or sohoots. 


Test PESPONSES Wil De Gobecied by NalCen Socal Research. Data wil be meinned win the National Pupil 
Endowment Foundaton'’s Data Archive wilin ig managed by the Flecher Family Trust and 1h anonymised 
form ip the UE Dela Archive. ‘Tour cic children's tata wil be treated with ihe cinicies! confidence. 
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Appendix B: Analysis plan 


Trial objective 
Primary objective 


To compare performance in English (reading) and mathematics between children whose parents 
were: 
e offered an incentivised place in the Parenting Academy; 


e offered an un-incentivised place in the Parenting Academy; and 


e not offered a place. 


Secondary objective 


To assess whether the estimate of impact differs when using alternative analysis approaches, in 
particular when considering the use of re-randomisation. 


Sample size 


Minimum detectable effect sizes were calculated assuming 2200 children across 15 schools, based on 
the number of pupils in the participating schools. The intra-class correlation coefficient was assumed 
to be between 0.05 and 0.2, and the explanatory power of covariates, at individual level only, between 
0 and 60%. Making these assumptions, the following minimum detectable effect sizes were calculated, 
expressed in standardised means, and given at the 5% significance level and 80% power: 


Values for intra-class correlation coefficient (rho) 


Unadjusted 


* proportion of residual variance at level 2 explained by covariates 


Note that these effect sizes are updated from the ones given in the original proposal. 
Randomisation 


The randomisation used a six-step process: 


1. From each family with more than one eligible sibling, one sibling was selected at random for 
assignment. If there existed a non-selected sibling in the same year group (either Year 3 /4 or 
Year 5/6), the non-selected sibling was given the same assignment (but unmatched — see 
below) as the selected sibling, otherwise assigned to the (unmatched) control group. 


2. Pupils were stratified by school, year, gender, English as additional language (EAL), ethnicity 
(White British versus Other), and whether or not additional data linkage consent was given. 
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3. Within each stratum, pupils were ordered by Key Stage 1 Average Point Score (KS1 APS, 
comprising of reading, writing and maths scores), and then paired consecutively with two other 
pupils to form a triplet. If the number of pupils in a stratum was not divisible by 3, one or two 
pupils were randomly removed to create adequately-sized strata. 


4. Pupils removed at step 3 were pooled; stratified by school, year and whether or not additional 
data linkage consent was given; and paired to triplets as described in step 3. If the number of 
pupils in a stratum was not divisible by 3, one or two pupils were randomly removed to create 
adequately-sized strata. The removed pupils were assigned to the (unmatched) control group. 
All triplets formed in steps 3 and 4 were pooled. 


5. By school, triplets were randomly assigned to one of three triplet assignment options (T1-T2- 
C; T2-T2-C; and C-C-C), reflecting the resource availability in each school for offering 
treatments 1 and 21°. Within each triplet, pupils were then randomly assigned the 
assignment options available. 


6. Steps 1-5 were carried out 1000 times, each time testing balance on the original stratification 
variables (school, year, gender, EAL, ethnicity, data linkage consent) plus five additional 
balance variables, using multinomial logistic regression. The additional balance variables 
were special educational needs status (SEN), eligibility for free school meals (FSM), term of 
birth, banding of the school’s location on the Index of Multiple Deprivation 2010 (IMD) and 
pupil attendance rate. The randomisation with the highest p-value was used for assignment. 


As a result of randomisation, the 2690 pupils available were assigned as outlined in Table 26. 


Table 26: Treatment assignment from randomisation 


A B C D E 
Groups 1-3 All cases All cases 
Alltriplets (incl. siblings (excl. pupils — (incl. pupils 
T1-T2-C (excl. insame year withmissing with missing 
NUMBERS assignment siblings) group) data) data) 
Incentivised 550 550 578 578 578 
Unincentivised 550 656 690 690 690 
Control 
(matched) 550 876 922 922 922 
Control 
(unmatched) 237 500 
1650 2082 2190 2427 2690 
Outcomes 


INCAS assessments in reading and maths, provided by the Centre for Evaluation and Monitoring at 
the University of Durham, will be used to test outcomes. Outcomes are only measured after the 
intervention was delivered, without baseline. Key Stage 1 outcomes in reading and maths will be used 
as baseline predictors. 


10 Every school offered at least as many T2 spaces as T1 spaces, hence there was no need to T1-T1-C triplets. 
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Analysis 


Analysis will be conducted on an intention-to-treat basis, including all children matched to groups 
(column C in Table 26). Analyses will be conducted in STATA version 12, using 2-sided significance 
tests at 5% significance level, both for comparisons between treatment and control and for any 
comparisons between treatments. 


Baseline characteristics 


Baseline characteristics (gender, age, Key Stage 1 reading and maths scores, ethnicity, EAL, SEN, 
FSM, attendance, IMD,) will be summarised by intervention group per year for each school and across 
all schools. Continuous variables (age, attendance) will be summarised with descriptive statistics (n, 
mean, standard deviation, range and median). Frequency counts and percentages of pupils by 
intervention group will also be provided by school and overall. 


Trial completion 


CONSORT diagram will be used to present summary of the flow of eligible children and their schools 
from recruitment through baseline assessment, randomisation, post intervention assessment and 
analysis. The number of children and schools included or excluded at each stage will be clearly stated 
and the reasons for exclusion will also be stated. 


Primary analysis 


The primary analysis will compare the two intervention arms with the control group using multilevel 
model to account for heterogeneity between schools. Unadjusted analyses with only pre-test scores 
and the intervention groups will be performed to investigate the impact of the stratification variables. 
However, effect sizes with 95% confidence intervals based on analysis adjusted for stratification 
variables will be reported as the main finding for this study. If either of the intervention groups is 
significantly different from the control group, comparison between the intervention arms will also be 
reported. 


Secondary analysis 


Secondary analysis based on a randomisation test will be performed to investigate the impact of 
balancing variables, matching and re-randomisation. The 2190 eligible children will be re-randomised 
to generate a matrix of 1000 acceptable re-randomisation runs. The acceptance criteria is p-value (no 
effect of the stratification variables) greater than 0.5. Under the null model of no treatment effects, 
post-test scores are expected to be independent of re-randomisation runs (Morgan and Rubin, 2012, 
p.3). Using the pre-defined re-randomisation matrix, we will perform: 


1. Simple unadjusted analysis between post-test scores and the intervention groups. We will 
report quantile based 95% confidence intervals for the average intervention effects across the 
re-randomisation runs. The proportion of re-randomisation results greater than the actual “trial 
intervention groups” will be reported as p-value. 


2. Adjusted analysis for the stratification and balance variables using a multilevel model. 95% 
confidence intervals and p-value will be calculated as described above. 


Further analyses will be performed to explore the influence of parental attendance at parenting 
classes. The degree to which attendance account for the effects attributed to the particular treatment 
will be estimated. This mediational analysis formally establishes the functional relationship between 
offering treatment and improved outcomes, ruling out influences attributable to other causes or simply 
chance. This type of analysis also allows the investigation of dose-response. Also, sensitivity analysis 
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for missing data will be performed by reported the distribution of missingness by intervention groups 
and the baseline factors. 


Finally, if one of the two treatment arms shows an effect but not the other, or if both show effects that 


are Statistically significantly different from each other, we will explore the impact of incentives on 
attendance. This will help explore the extent to which incentives mediate outcomes via attendance. 


58 


Parent Academy 


Appendix C: Cost rating 


Cost ratings are based on the approximate cost per pupil per year of implementing the intervention over 
three years. Cost ratings are awarded using the following criteria. 


Cost Description 
£ Very low: less than £80 per pupil per year. 


£E Low: up to about £200 per pupil per year. 
EEE Moderate: up to about £700 per pupil per year. 
ELLE High: up to £1,200 per pupil per year. 
ELLEEE Very high: over £1,200 per pupil per year. 
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Appendix D: Padlock rating 


7 July 2016 Complete by Elena Rosa Brown 


2. Power: 5. Threats to 
1.Design: 3. Attrition: 4. Balance: RET ae 
What is What is the VElitlia’e 


What is the Adjustment 


minimum Adjustment to 
quality of the (\V-) Me) el ge) e) to rating to 
fol} ¢-eie-]e) (= account for issues 
design of the out from the account for 
uation? effect at the iatienD balance? with 
evaluation? valuation: ance? 
: start? bette - m interpretation? 
Evaluation design Implementation Analysis and interpretation 


Rating 1. Design 2. Power 3. Attrition 4. Balance LoYa Mal aysleom ce) 
(Le) =59) WEN CoTi a 


Fair and clear experimental 
design (RCT, RDD) . 


Well-matched comparison 

(quasi-experiment) : < 30% 
Matched comparison 

(quasi-experiment) . < 40% 
Comparison group with 

poor or no matching . < 50% 
No comparator 0 Imbalanced on re 


The final security rating for this trial is 4 @. 
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You may re-use this document/publication (not including logos) free of charge in any format or 
medium, under the terms of the Open Government Licence v2.0. 


To view this licence, visit www .nationalarchives. gov.uk/doc/open-government-icence/version/2 or 


email: psi@nationalarchives.gsi.gov.uk 


Where we have identified any third party copyright information you will need to obtain permission 
from the copyright holders concerned. The views expressed in this report are the authors’ and do 
not necessarily reflect those of the Department for Education. 


This document is available for download at wwaw.educationendowmentfoundation.org.uk 
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